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What Has Made ROI for CTE Difficult? 


Career and technical education (CTE) is increasingly seen as a major potential contributor to the 
recovery of the U.S. economy. However, the effectiveness and impact of the current Carl D. 
Perkins Act (otherwise known as Perkins IV) that governs CTE are unclear. Is the federal 
investment in CTE paying off? To answer this, we need to establish the internal efficiency of 
CTE by comparing the costs and benefits of implementing CTE using Perkins funds at different 
enterprise levels. 1 A second question is whether CTE has a measurable impact beyond the 
enterprise level at which it is being implemented. This question focuses on external effectiveness. 
Answering these questions may put to rest the frequently held notion that CTE — and by 
association Perkins IV — has been largely ineffective in affecting U.S. education and workforce 
development (Duncan, 2011). As shorthand, this report will refer to the measurement of the 
internal efficiency and external effectiveness of Perkins IV and CTE as the return on investment 
(ROI) for CTE; the report will also describe other related tools that generally fall under the 
broader rubric of program evaluation (Priest, 2001). 

Examining the ROI for CTE has been difficult for many reasons. To begin with, there is weak 
connectivity within CTE between the three elements that are needed to conduct ROI: data and 
measurement, accountability and evaluation, and research. CTE data and measurement are 
collected primarily to meet accountability requirements under Perkins IV. Systems for data and 
measurement concerning CTE are generally discussed at state and federal government levels to 
ensure compliance with federal Perkins accountability requirements. Much effort at both the 
national and state levels has gone into standardizing variable definitions and measurement 
approaches with the goal of developing a common set of CTE data (Schray, 2000). Although 
CTE has not achieved full standardization when it comes to data and measurement, much 
progress has been made over the last decade under the leadership of the Office of Vocational and 
Adult Education (OVAE) of the U.S. Department of Education. Such efforts include the 
extensive work of multiple national Data Quality Institutes (DQIs) as well as the Next Steps 
Working Group (NSWG), an OVAE-sponsored monthly electronic town-hall meeting on issues 
affecting Perkins and CTE accountability. Both the DQI and the NSWG meetings were initiated 
and led by OVAE to establish consistency across different data terms and measurement 
definitions. 2 

More recently, the NSWG has studied the inclusion of secondary CTE measures and their 
alignment with those available within the Education Data Exchange Network (EDEN). This 
group has discussed how Perkins data should be appropriately linked, connected, and 
incorporated within the EDEN database. The NSWG has reviewed, and is reviewing, the 
technical specifications used for Perkins secondary CTE, 3 with a particular emphasis on those 


1 This report uses the word enterprise genetically to refer to the activities, programs, and services that take place at 
the different levels at which CTE functions and operates. This includes functioning and operating at the state level, 
at sub-state levels such as school districts or postsecondary institutions, or at sub-system levels such as classrooms 
in a high school or sub-units at a community college (see Connecting CTE to ROI Measures). 

2 Full details related to the DQI and the NSWG efforts can be found at http://cte.ed.gov . 

3 EDEN contains a list of technical specifications that is available at 

http ://w ww2 , ed.gov/ about/ inits/ed/ edfacts/ index. html . See also http://cte.ed.gov for more information about the 
NSWG discussion regarding CTE-EDEN linking. 
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measures relating to achievement on academic proficiency tests, high school completion, and 
graduation. The hope for these increased interactions between the NSWG and EDEN is greater 
consistency in CTE accountability data across the nation and in academic comparisons of CTE 
and non-CTE students. Because of definitional, technical, and policy differences in the way CTE 
and EDEN define academic and technical achievement at the high school level, much work 
remains if the field is to achieve consistency in CTE data and align such data with other U.S. 
Department of Education initiatives on data gathering like EDFacts. 4 

The Postsecondary Common Data Dictionary project, 5 a National Research Center for Career 
and Technical Education (NRCCTE) project jointly led by staff from the NRCCTE and from 
MPR Associates, Inc., also sought to address the extent to which a common data dictionary 
would yield more consistent, valid, and reliable state and national postsecondary accountability 
measures (Kotamraju, Richards, Wu, & Klein, 2010). A critical finding of this project was that 
participating states were able to uniformly define many data elements that can be used to 
construct and report on postsecondary accountability measures required by Perkins. Guided by 
participating states, the NRCCTE Postsecondary Common Data Dictionary project team selected 
data elements that would become the foundation for building an accountability system when 
Perkins is reauthorized in the future. The project acknowledged differences in how states collect 
data, but a common data dictionary provides a standard goal to which all states can link their 
own data. Using a common data dictionary, and with sufficient support, many states should be 
able to generate substantially similar information for Perkins accountability. 

Lack of a Uniform Global CTE Database for Accountability and Evaluation 

Despite partial success at moving toward standardization of data and measurement, CTE has 
made little progress toward achieving the ideal of a unifonn, global database for accountability 
and evaluation, a goal first discussed in the early 1990s when the Perkins II legislation was 
enacted. However, CTE is not alone in this regard. An ideal framework for accountability and 
evaluation within the education sector has been difficult to achieve, although many have tried to 
establish one (Ewell & L’Orange, 2009). Even today, education data for accountability and 
evaluation are obtained from three disparate and generally unconnected sources: (1) state-based 
unit-record data systems, (2) national data systems such as EDEN and the Integrated 
Postsecondary Education Database System (IPEDS), and (3) National Center for Education 
Statistics (NCES) Sample Survey Data. Each of the above data sub-systems requires 
considerable technical expertise; each is also limited in the extent to which it could serve as a 
common accountability and evaluation system. A key point about existing national databases 
(particularly NCES and IPEDS) and state-level education databases (secondary and 
postsecondary) is that in the development, maintenance, and usage of such databases, CTE has 
not been a key focal point and as yet has had limited participation. 

Should the data contained in the above three data sub-systems be merged into a single, national 
database, they would need to be housed in a nationally agreed-upon central location — an unlikely 


4 See the following two websites for more details: EDFacts at http://www.ed.gov/open/plan/edfacts and EDEN at 
http://www2.ed.gov/about/inits/ed/edfacts/overview.html . 

5 See http://l 36. 1 65 . 1 22. 1 02/mambo/content/view/53/ . 
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scenario given current tight budgets. Also, states and the sub-systems within them (e.g., 
postsecondary institutions and school districts) have not been in favor of submitting unit-record 
data to a national system. Additionally, state data privacy and Family Educational Rights and 
Privacy Act (FERPA) regulations make a central database problematic for most state and local 
education agencies (see Kanstoroom & Osberg, 2008, who devote several chapters to FERPA). 
Further, large, longitudinal national and state educational databases have not been without 
controversy. Nevertheless, the U.S. Department of Education funding of the State Longitudinal 
Education Data Systems (SLDS) is one such effort to help states better manage and use student 
educational data (Data Quality Campaign, 2009). CTE’s contribution to developing the SLDS — 
or a P-20 database system, which many states are developing in conjunction with SLDS — is still 
an open question. 

Those working in the field of CTE accountability and evaluation have grappled with the need for 
uniform, global information — a national database — to meet CTE’s multiple needs, including 
career guidance, accountability and evaluation, and program improvement. At present, the 
United States has no national, comprehensive database that meets the accountability 
requirements prescribed in the Perkins IV legislation. Individual state-level databases that collect 
CTE accountability information do exist, but these generally have been built to serve state- 
specific purposes and requirements. Most have little or no connection either to other databases 
within the state (most states keep their secondary and postsecondary CTE data separate) or to 
those of other states, let alone to a national system. For example, the United States currently has 
at least 54 different state-based CTE data systems 6 (or at least 108, given the states’ separate 
secondary and postsecondary CTE data systems), a less than optimal situation for a common 
accountability framework (Kotamraju, 2012). 

Other Difficulties Facing ROI for CTE 

Too often, CTE treats accountability and evaluation synonymously. Accountability usually 
means the achievement of a set target, whereas evaluation implies testing how well the 
achievement of the target meets overall purposes and goals (e.g., meeting the targets outlined in 
Perkins IV). Evaluation is generally tied to research, and research is usually conducted at 
universities and research and policy organizations. These institutions’ limited access to linked 
education and workforce datasets housed within state and local agencies makes conducting ROI 
studies difficult. Moreover, for many research-focused organizations other than the NRCCTE, 
the systematic evaluation of how well CTE is doing is still a very small portion of their overall 
work, further limiting the scope for ROI for CTE. 

Another difficulty related to conducting ROI for CTE is that institutional research (IR) capability 
within CTE often lacks the degree of sophistication necessary for undertaking ROI studies. 
Seymour, Kelly, and Jasinki (2004) offered the following definition of institutional research that 
can be modified for CTE. “Institutional research involves the collection of data or the making of 
studies useful or necessary in (a) understanding and interpreting of the institution; (b) making 
intelligent decisions about current operations or plans for the future; and (c) improving the 

6 This number includes the four U.S. territories — the District of Columbia, the U.S. Virgin Islands, Puerto Rico, and 
Guam — to which Perkins funds are distributed. 
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efficiency and effectiveness of the institution” (p. 54). This definition describes the skills 
necessary for conducting ROI studies, and these skills are in short supply within the CTE field. 

Frequently, the process of building the intricate connections between inputs, process measures, 
outputs, and outcomes — a prerequisite for conducting ROI — is difficult (Peterson & Augustine, 
2000; Yorke, 2004). Further, these connections are generally missing as an IR function (Seymour 
et ah, 2004). This is particularly true for CTE. Most of the time, IR expertise at CTE offices 
within local eligible agencies (LEAs) and state eligible agencies (SEAs) is assigned to meeting 
immediate and short-term accountability requirements rather than medium- and long-term 
evaluation studies of how well CTE is doing. In other words, IR in CTE is perceived simply as 
data collection and reporting. This issue is not exclusive to CTE, but applies to much of 
education as well (Ewell, 2002; Ewell & Boeke, 2007; Serban, 2002; Volkwein, 1999, 2008). 

ROI for CTE, and maybe even for all of education, has remained limited because of the general 
perception among educators, particularly in CTE, that ROI is a business-like technique primarily 
concerned with money and finances that is not relevant to teaching and learning. Teaching and 
learning are considered the core of academics, 7 whereas services outside of teaching and learning 
are deemed the periphery (Toma, 2007). As a result, ROI for education is conducted only on the 
periphery because the periphery contains items that can be measured, such as tuition and fee 
revenues, revenues from specialized and targeted services, expenditures on student services, and 
infrastructure expenditures. The key to education’s success — teaching and learning — is often 
excluded from ROI because it is considered non-measurable. Full development of the 
frameworks, processes, and procedures for conducting CTE for ROI must include the core of 
CTE — teaching and learning — and indicate how well this core is doing. More importantly, if 
ROI for CTE is to be optimally conducted, the CTE core must be linked to student, program, and 
state perfonnance. 

An Overview of This Report 

After scanning the literature on ROI approaches and methods and their application to CTE, the 
NRCCTE developed a report (Stone, Kotamraju, Aliaga, & Blackman, 2010) that clearly 
indicated that a full-fledged ROI for CTE study is both cost prohibitive and challenging for the 
reasons already outlined. Nevertheless, there remains a need to infonn the CTE community 
about the building blocks that are needed for conducting ROI for CTE. As a result, the 
development of this report became a sub-project within the NRCCTE ’s overarching CTE 
Accountability and Evaluation: a Comprehensive Strategy for Technical Assistance project. 

This report takes a global approach to conducting ROI for CTE. It provides a broad primer of 
what ingredients need to be taken into consideration in ROI studies. The Building Blocks for ROI 
Studies outlines the building blocks for conducting ROI studies. Connecting CTE to ROI 


1 Academic here refers to the traditional educational structural elements and practices — like curricula, syllabi, 
coursework, methods, testing, and grading — that take place within secondary and postsecondary institutions. This 
definition is different from what sometimes is used in CTE, where the use of the term academic refers to the set of 
core courses (e.g., English, Math, Science, Social Studies) that high school students must take to meet high school 
graduation requirements (Nord et ah, 201 1). 
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Measures shows how these building blocks are connected to CTE. Setting the Stage for 
Measuring ROI for CTE discusses different approaches for measuring ROI for CTE and provides 
a basic protocol that CTE might adopt when undertaking ROI for CTE. Appendix A provides 
abstracts of ROI studies across the nation. These studies are organized state by state; for each 
state, Appendix A identifies which ROI for CTE approach was used and at what level within the 
state the study was undertaken. Other appendices provide additional information the field might 
consider when choosing to follow the common protocol in conducting ROI for CTE. 
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The Building Blocks for ROI Studies 


ROI studies have many shapes, sizes, and forms. Regardless, all have one thing in common — the 
relationship between a set of benefits and a set of costs that are associated with any enterprise 
(Phillips & Phillips, 2008). This section of the report addresses how these benefits and costs are 
counted and accounted for, in what context benefits and costs are juxtaposed, and what is 
obtained when benefits and costs are connected in particular ways to one another. First, this 
section discusses the raw materials of an ROI study. Second, it outlines different approaches to 
an ROI study. Third, it discusses the steps required for developing an integrated logic model that 
forms the rationale for an ROI study. 

ROI Terminology 

Developing ROI for any enterprise begins with identifying indicators. An indicator is the basic 
raw material used in program evaluation, of which ROI is just one method (Priest, 2001). 
Indicators include inputs, process measures, outputs, and outcomes (Burke & Minassians, 2002, 
2004). Inputs involve the human, financial, and physical resources received to support programs, 
activities, and services, examples of which include funding, enrollments, and staffing indicators. 
Process measures are the means used to deliver programs, activities, and services — means such 
as assessment of student learning, use of technology, and teacher training. Outputs reflect the 
quantity of products actually produced, such as the number of degrees awarded, the number of 
majors in a program, the number of students who have transferred to other institutions, and the 
number of students who have graduated. Outcomes cover the quality of programs, activities, and 
services, and their benefits to students, states, or society. Common outcome measures in 
postsecondary CTE include retention, graduation, and transfer rates, time to degree, test scores, 
and job placements. 

Inputs and outputs, not process measures and outcomes, dominate the attention of the education 
community. With their emphasis on quantity, inputs and outputs are easy to count, measurable, 
and easy to compare. On the surface, they are easier to assess and understand, are often taken for 
granted, generate less controversy within the education community, and, most important, are 
within the community’s direct control. They are contextual because they can be defined as 
outputs at one level (e.g., the high school level), but as inputs at another level (e.g., the college 
level). 

Process measures and outcomes flow out of inputs and outputs, creating an integrated system 
(see below). For instance, Volkwein (1999) has argued that inputs matter because the “number of 
extremely important outcomes are highly predictable from the inputs. We can predict about one- 
third of the variance in student academic performance and two-thirds of the variance in 
persistence to graduation by knowing the high school rank in class and SAT scores of freshmen” 
(p. 13). Here high school rank in class and the SAT scores of freshman can be viewed as inputs 
with college graduation as the output; on the other hand, the same high school rank in class and 
the SAT scores of freshman are outputs at the high school level that could be related to inputs 
such as high school GPA or the number of days absent from high school. 
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Though well understood within the education community, process measures are harder to 
comprehend by constituencies outside education. They are generally the means by which 
educators conduct their business. But because they generally reflect non-measurable qualitative 
elements in an enterprise, external authorities have a harder time understanding them and as a 
result may decry them because of their vagueness and lack of comparability across various 
(quantitative) units of measurement (Burke & Minassians, 2002). However, within an ROI 
framework, as indicators, process measures are critical because they provide a context for 
“quantifying” what generally is non-measurable. 

Outcomes represent policy values, which are both elusive and subjective (Hubbell, 2007). They 
are usually expressed as questions: Are students learning well? Are clients in a training program 
getting jobs? Are faculty members responsive to students? Is the institution serving the 
community appropriately? Outcomes are sometimes subsumed under the heading of outputs, but 
distinguishing the two as separate measures is critical when policy values (e.g., efficiency, 
equity, choice, or quality) are being discussed. Speaking broadly, efficiency represents the value 
added; equity addresses the issues of access and affordability; choice reflects targeting a specific 
range of options; and quality implies exceeding or attaining a prescribed perfonnance standard 
(Burke & Minassians, 2004). Although measuring outcomes is difficult, it is necessary because 
outcomes are the only true measure of ROI. 

o 

ROI as a Numerical Value 

ROI usually takes a balance-sheet approach, chalking up benefits and costs that follow a 
particular protocol, as shown in Figure l. 9 Benefits and costs themselves are divided into private 
(internal to the enterprise) and social (external to the enterprise) categories. Benefits and costs 
must include the monetized values of all non-monetary benefits and costs to fully measure the 
internal efficiency and external effectiveness of an enterprise. 


s The information presented in this sub-section has been simplified for ease of understanding. For a more technical 
understanding, readers are referred to Hollenbeck (2011), Johnstone (2008), Shively and Galopin (n.d.), and World 
Bank (2007). 

9 The common protocol for ROI for CTE is discussed in more detail in Setting the Stage for Measuring ROI for 


CTE.. 


7 




Figure 1. ROI terminology: How does it all connect? 

The different items, and the way they connect to each other, listed in Figure 1, leads to the 
calculation of five things: (a) the opportunity cost — a measure of what is being given up in order 
to undertake the activity, used by quantifying costs and benefits, (b) the time horizon — or how 
long the enterprise will be in place, when benefits will begin to be observed and fully realized, 
and when costs will begin to appear and accrue, (c) the discount rate — the appropriate rate that 
needs to be applied to convert the value of future costs and benefits to the present time, (d) 
monetization — the translation of all non-monetary benefits and costs into monetary values, and 
(e) externalities — the measurement of the negative and positive impacts of all monetary and non- 
monetary benefits and costs that result from having the enterprise in place. When any or all of 
these calculations are omitted, the result is invariably the under- or over-estimation of costs and 
benefits, which in turn skew the assessment by skewing the measures that are described next. 

Broadly speaking, ROI is reflected by numbers: these include the benefit cost ratio (B/C; a 
number greater than one implies that the enterprise is justified on both internal efficiency and 
external effectiveness grounds); the net present value (NPV; a number greater than zero implies 
that building the enterprise today instead of waiting for the future is justified); and the internal 
rate of return (IRR; when the rate of return obtained from enterprise implementation exceeds the 
market interest rate, 10 thereby making it worthwhile). 


10 Strictly speaking, the rate to which the IRR is compared is called the discount rate , the choice of which can vary 
depending on the particulars of the project that is being considered for implementation (Shively & Galopin, n.d.) 


An Integrated Logic Model for ROI 

At one level, ROI is easy to understand. As indicated, once all measurable and non-measurable 
benefits and costs are counted and accounted for, ROI is expressed as a numerical value that 
indicates the worthiness of an enterprise. At the same time, assessing ROI is difficult because it 
is contextual and subjective. It comes at the logical end of a series of steps that are undertaken as 
part of a fully integrated program evaluation model (Priest, 2001). ROI can be placed within an 
integrated logical model that connects the five focal points, answering each question 
sequentially, and collects information on the four key indicators — inputs, process measures, 
outputs, and outcomes — from which costs and benefits are derived and a corresponding program 
evaluation metric is obtained. Underlying the entire logic model are scientifically based research 
methods, which when implemented properly generate a comprehensive program evaluation. A 
well-known program evaluation handbook (W. K. Kellogg Foundation, 2004) summarized the 
connection between focal points, key indicators, and metrics: “A program logic model is a 
picture of how your program works — the theory and assumptions underlying the program. . . This 
model provides a road map of your program, highlighting how it is expected to work, what 
activities need to come before others, and how desired outcomes are achieved” (p. 35). Figure 2 
shows this program logic model for ROI. 


Needs Assessment Feasibility Study Process Evaluation Outcome Evaluation S Cost Analysis 
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Figure 2. Program logic model for ROI. 

In modifying the original table found in Priest (2001), and by adding information from Phillips 
and Phillips (2008), Table 1 provides a framework for different aspects of program evaluation. 
The five focal points that are summarized in the table are taken from Priest (2001). Each of the 
five focal points addresses a primary question: 
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Table 1 


Comparison of the Five Kinds of Program Evaluation 



Needs 

Assessment 
(Focal Point 1) 

Feasibility 

Study 

(Focal Point 2) 

Process Evaluation 
(Focal Point 3) 

Outcome 
Evaluation 
(Focal Point 4) 

Cost Analysis 
(Focal Point 5) 

Measures 

Gap between 
what is and 
what should be 

Alternative 

approaches 

help/hinder 

factors 

Gap between program 
plan and execution 

Satisfaction levels, 

objectives 

attainment 

Comparative 

merit/worth 

Questions 

Asked 

What are 
objectives, 
priorities, and 
needs? 

Which 

strategies and 

program 

procedures? 

Are strategies and 
procedures working? 

Are objectives 
met? Are people 
satisfied? 

Should program 
be continued? 

Get Input From 

Clients, 
customers, and 
community 

Staff, supplier, 
clients, and 
customers 

Staff, supplier, clients, 
and customers 

Clients, customers, 
and community 

Staff, supplier, 
and profession 

Answers Used 

To 

Understand 
context and 
direct planning 

Gauge viability 
and best use of 

resources 

Monitor and modify 
program (midcourse) 

Improve/justify 

effectiveness 

Decide on future 
offerings 

Results Used 

By 

Staff and 
supplier 

Staff, supplier, 
and customer 

Staff and supplier 

Staff, supplier, 
clients, and 
customers 

Staff, supplier, 
and customer 

Conducted By 

Describing 
context and 
comparing 
actual 

circumstances 
with intended 
change state 

Inventorying 
any resources or 
barriers and by 
examining all 
realistic 
possibilities 

Comparing arising with 
anticipated need and 
content or format with 
intended design 

Comparing actual 
result or product 
with expected 
outcome or 

standard 

benchmark 

Comparing cost 
($) with benefit, 
effect, utility, 
and efficiency 

Other 

Considerations 

Goals vs. 

objectives, 

unused 

opportunities, 

underlying 

problems, and 

unrealized 

needs 

Identify legal, 
moral, political, 
and fiscal 
restrictions, 
support, 
constraints, or 
limits 

Identify weaknesses and 
strengths, remain 
flexible, and suggest 
quick adjustments 

Baselines measures 
may need to be 
taken if trying to 
measure change 
over time 

Compare with 
other programs, 
repetition ease, 
subjective value 

Related Forms 
of Evaluation 
(Also Known 

As) 

Context 

evaluation, 

objectives 

evaluation, 

demand 

evaluation, 

discrepancy 

evaluation 

Input 

evaluation, 

planning 

evaluation, 

practicality 

evaluation, 

comparative 

evaluation 

Formative evaluation, 
progress evaluation, 
implemented evaluation, 
transactional evaluation 

Summative 
evaluation, impact 
evaluation, product 
evaluation, 
performance 
evaluation 

Cost-benefit 
analysis, cost 
effectiveness, 
cost 

minimization- 

utility 

maximization, 
cost efficiency 

Program 

Evaluation 

Metric 

Reaction and 
perceived value 

Learning and 
confidence 

Application and 
implementation 

ROI: Impact and Consequences 

External 

Effectiveness 

Internal 

Efficiency 


Source. Phillips and Phillips (2008); Priest (2001). 
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• Needs Assessment : What are some of the gaps the program will fill? 

• Feasibility Study. Given certain constraints, can the program succeed? 

• Process Evaluation : How is the implemented program progressing? 

• Outcome Evaluation : Were program goals and objectives achieved? 

• Cost Analysis'. Was the program financially worthwhile or valuable? 

Moreover, ROI is just one piece of available infonnation that is used in conjunction with other 
types of infonnation obtained at the different focal points at which program evaluation takes 
place. These may include reaction and perceived value (Focal Point 1), learning and confidence 
(Focal Point 2), application and implementation (Focal Point 3), ROI (Focal Point 4), and impact 
and consequences (Focal Point 5). 11 The five focal points are similar to, if not the same as, the 
five kinds of program evaluation identified by Priest (2001; see Table 1). 

Program logic models are developed to show the work that is planned and the results are 
intended or expected. The logic model identifies appropriate and relevant success measure 
should the planned work be completed and the intended results actualized. The success measure 
identified in Figure 2 may be any of those indicated in the last row of Table 1. In essence, to 
implement the program logic model fully, the process needs to be repeated for each focal point 
identified in Table 1. Answering all items related to each row in Table 1 for each focal point 
leads to a specific success measure for that focal point. Because this report focuses on ROI, the 
relevant success measures are under Focal Points 4 and 5. Note that the success measures for 
each focal point shown in Table 1 are not discrete but are instead continuous and comprehensive. 
The various success measures flow into each other, and if one is used separately with no 
reference to the others, it will provide only partial and perhaps even misleading information. 
When taken together, the program evaluation metrics provide information regarding internal 
efficiency and external effectiveness. 

Balancing Internal Efficiency and External Effectiveness When Conducting ROI 

ROI frameworks, processes, procedures, and tools are not developed in a vacuum. Often the 
choice of frameworks, processes, procedures, and tools depends on the balance that is struck 
between internal processes of the enterprise and the external pressures placed upon it (Alfred, 
2008). Internal processes of the enterprise relate to measuring internal efficiency; 
correspondingly, measuring external effectiveness relates to external pressures placed on the 
enterprise. Internal efficiency is driven by cost considerations, whereas underlying external 
effectiveness is a notion of quality (Fretwell, 2003; Hummel-Rossi & Ashdown, 2002). 
Sometimes quality gets confused with efficiency, although it is to effectiveness that quality 
relates most (Hubbell, 2007; Volkwein, 1999). 

So ROI is essentially a quantitative measure that represents something of value. But what is this 
something of value? Hubbell (2007) defined it as quality, related to two other concepts, 
efficiency and accountability. In Hubbell’s words, efficiency “is the barometer of the how of 


1 1 Added into this mix are intangibles that create context and subjectivity and that need to be included as part of the 
comprehensive program evaluation (Phillips & Phillips, 2008). 
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operations (and) looks inward and asks if we are doing the right work, crisply and well — 
strategic outcomes, effectively delivered, at appropriate quality levels. Efficiency requires that 
the work be maximally cost-justified. Accountability is tied to stewardship with responsibility 
for creation and use of resources and a public reckoning of how they are used” (Hubbell, 2007). 
In this report the term internal efficiency is equivalent to Hubbell’s concept of efficiency. The 
term external effectiveness, as used in this report, is what Hubbell called accountability. Taken 
together, internal efficiency and external effectiveness jointly measure the value — as defined by 
the ROI — of any enterprise. 

Ideally, an enterprise should undertake both internal efficiency and external effectiveness in 
calculating an ROI. However, in some situations and circumstances, one is emphasized more 
than the other. Regardless, both internal efficiency and external effectiveness require ensuring 
that Focal Points 1, 2, and 3 (see Table 1) are completed first. Only then will there be a need to 
address Focal Points 4 and 5, as well as the balance that needs to be struck between these two 
focal points. Further, the choice of emphasis between Focal Point 4 and Focal Point 5 depends on 
the enterprise level at which the ROI is conducted. We look at this more deeply within the 
context of CTE in Connecting CTE to ROI Measures. 
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Connecting CTE to ROI Measures 


So far, our discussion has centered on how ROI is implemented. However, CTE is undertaken on 
at least three different enterprise levels: at the overall state level; in subsystems such as the 
secondary (e.g., high school or school district) or postsecondary (e.g., two-year college) levels; 
and at the specific program level within high schools, school districts, or postsecondary 
institutions. Figure 3, below, shows how these three levels connect within CTE. 

A key question arising from Figure 3 is how the information available within CTE can be used to 
develop the building blocks of ROI (the left-hand side of the graphic) so that measures of 
internal efficiency and external effectiveness can be determined. This section describes the three 
different techniques researchers and practitioners apply when conducting ROI studies. First, this 
section discusses the reasons for and purposes behind conducting ROI for CTE at the three 
different enterprise levels. 

Reasons for Conducting ROI for CTE 

Typical reasons for conducting ROI are (a) meeting accountability requirements, (b) program 
improvement, and (c) marketing. Meeting accountability requirements implies the need to 
meeting program objectives; making better planning decisions regarding programs; authorizing 
fiscal payments; meeting grant obligations; and allocating resources correctly. Program 
improvement means identifying strengths and weaknesses, creating safer practices, increasing 
educational value, enhancing competence, testing innovative ideas, reducing planning problems, 
decreasing operating costs, reducing staff concerns, and establishing quality benchmark and 
assurance standards. Marketing involves advertising past program effectiveness, indicating 
successful program track records, promoting public relations, and advocating and lobbying for 
maintaining and sustaining policy (National Association of State Directors of Career Technical 
Education Consortium [NASDCTEc], 2010). As will be explained below, this definition of 
marketing fails to taken into the social good (Kotler & Zaltman, 1971), which is the basis for 
conducting ROI in the first place. 

Different enterprise levels (states, schools, colleges, and programs) regularly provide information 
on meeting accountability requirements, program improvement, and marketing. Information on 
meeting accountability requirements and program improvement is available in the Consolidated 
Annual Report (CAR) submitted by each state to OVAE at the end of each year. 12 Of the three, 
meeting accountability requirements takes precedence in the CAR, followed by program 
improvement, and, if at all considered, marketing. Also, OVAE submits an annual Report to 
Congress in which individual state information from the CAR is aggregated to provide a national 
picture of the extent to which accountability requirements under the Perkins legislation are being 
met (U.S. Department of Education, Office of Vocational and Adult Education, 2010). However, 
neither the individual state CAR information nor the national Report to Congress makes direct 
references to ROI for CTE measures. However, with some additional analysis using the 
information in the CAR (particularly around program improvement and financial data), 


12 See http://cte.ed.gov. 
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Figure 3. Connecting CTE plans, programs, budgets, and accountability to ROI measures. 
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ROI for CTE measures could be imputed at the state level. 1 3 

Marketing has only infrequently been used as a reason for conducting ROI for CTE. This is 
primarily because the restrictions on using state and federal funds for advocacy are stringent, and 
particularly given the fuzzy line between marketing and advocacy. However, there is a broader 
reason why marketing a social good like CTE has been limited, and that is because it is seen by 
many as crassly commercial. In the context of ROI for CTE, commercial marketing has not been 
clearly distinguished from social marketing. The former is discussed mainly in tenns of 
advertising, whereas the latter is connected more closely to strategic planning (MacFadyen, 
Stead, & Hastings, 1999). In their seminal article, Kotler and Zaltman (1971) defined social 
marketing as “the design, implementation and control of programs calculated to influence the 
acceptability of social ideas and involving considerations of product planning, pricing, 
communication, distribution and marketing research” (p. 5). This definition relates directly back 
to different kinds of program evaluation, including ROI (see Table 1 and associated discussion). 
In other words, with both the internal efficiency and external effectiveness of CTE increasingly 
coming under question, using social marketing principles as a basis for conducting ROI for CTE 
would be appropriate, particularly for the purposes of highlighting past program effectiveness 
and substantiating successful program track records. 

Purposes of Conducting ROI for CTE 

An ROI for CTE study has three purposes (see Table 2). The first is to make rational decisions, 
such as determining whether investing in a specific CTE enterprise is worthwhile — that is, 
determining whether its benefits exceed its costs. A second purpose is to make informed choices, 
such as choosing from among an array of similar CTE enterprises — at the school district or 
community college level — that provide varying benefits for more or less the same cost. A third 
purpose is the validation of strategic planning at the state level; for example, assessing the direct 
and indirect economic and social impact of allocating an annual amount for CTE in the overall 
state budget. Table 2 uses three examples to distinguish the three purposes of ROI for CTE and 
lists typical questions that must be answered if the various steps in the process (as outlined in 
Table 1 and Figure 3) are to be followed accurately. 

The infonnation presented in Table 2 should be regarded as a starting point in building an ROI 
for CTE process. Readers of this report are encouraged to develop additional questions that 
supplement those listed in Table 2. How these questions might be developed may be gleaned 
from the abstracts of ROI studies of different types presented in Appendix A, which describes in 
summary form how different states have approached conducting ROI for CTE. 

Working from right to left in Table 2 reveals an instructive point. Focusing on the third purpose 
(validating strategic planning) includes the second (informed choice) and the first (rational 
decision making) purposes as part of the overall ROI for CTE effort; similarly, when focusing on 
the second (informed choice) purpose, the first (rational decision-making) is included as well. 


13 As explained in the introduction, the Perkins accountability system is not a national system. Therefore, imputing 
national ROI for CTE measures is problematic (Kotamraju, 2012). 
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Table 2 

Purposes for Conducting ROI: A Progressive Questioning Approach 


Purposes 

Program Level: Rational 
Decision-Making 

Sub-System Level: Informed Choice 

State Level: Validating 
Strategic Planning 

A local CTE administrator in 
a state called Anywhere is in 
charge of running several 

CTE allied health programs 
within a newly constructed 
Healthcare Simulation 

Center. The CTE 
administrator receives 
$10,000 to buy state-of-the- 
art hospital simulation beds. 

Anywhere’s State CTE 
director asks the local CTE 
administrator to detail the 
program evaluation 
methodology used in 
justifying the purchase. 

Typical questions include: 

1 . How will the program 
evaluation be conducted 
and the corresponding 
metrics be developed? 

2. If the purchase is yet to 
be made, how will the 
program evaluation 
change? 

3. If, on assessment, the 
full impact of the 
purchase is limited, 
how would you modify 
your program 
evaluation? 

In your role as the State CTE Director in 
the state of Anywhere, you must choose 
among three innovative ideas that rose to 
the top when local consortia submitted 
different ideas through a competitive 
grant program for $50,000 in Perkins 
Leadership Funds. 

The three choices are: 

A. A statewide Healthcare Simulation 
Center that uses state-of-art 
technologies for all of the state’s 
allied health care programs. 

B. A statewide innovative curriculum 
strategy that builds communities of 
practice among academic and CTE 
teachers, and which infuses 
academics into the teaching of CTE 
courses and programs. 

C. A statewide effort to build capacity 
that places CTE measurement, 
accountability, and evaluation within 
the larger statewide data system. 

The focus of the grant involves aligning 
different policy frameworks, developing 
managerial oversight and administrative 
knowledge, connecting different data 
systems, and expanding institutional 
research expertise. 

Typical questions include: 

1 . What program evaluation strategies 
were conducted and what 
corresponding metrics were 
developed to make the final choice? 

2. Under what conditions does 
Anywhere’s State CTE Director need 
to go beyond quantitative 
information to consider qualitative 

The state of Anywhere is 
currently running a budget 
deficit. The Governor of 
Anywhere has requested a 
thorough review of all 
programs including CTE. 
Specifically, Anywhere’s 
Governor was interested in 
examining the economic 
impact of secondary and 
postsecondary CTE on the 
individual and on the 
economy in the state of 
Anywhere. He requested the 
University of Anywhere to 
develop various program 
evaluation metrics, but 
specifically asked them to 
focus on cost-benefit and net 
impact analyses. 

The primary research 
questions addressed by this 
study include: 

1 . How many program 
completers and graduates 
from secondary school 
programs, technology 
centers, and community 
college career and 
technical programs were 
employed in Anywhere in 
the quarters after 
graduation? 

2. What are the economic 
impacts of dual 
enrollment and 
articulation on the 
individual and the state? 

3. What are the earnings of 
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factors? 


program completers and 
graduates from secondary 
school programs, 
technology centers, and 
community college career 
and technical programs 
who were employed in 
Anywhere in the quarters 
after graduation? 

4. In what industries are 
graduates employed? 

5. What is the economic 
impact of those earnings? 

6. What is the economic 
impact of institutional 
operating expenses and 
employment in secondary 
and postsecondary career 
and technical education 
programs in schools, 
technology centers, and 
community college career 
and technical programs? 

7. What is the return on 
investment in secondary 
and postsecondary CTE 
programs in schools, 
technology centers, and 
community college CTE 
programs? 

Describe the analysis the 
University of Anywhere 
undertook to complete the 
study, remembering that it 
had to be designed to 
provide simple, 
straightforward answers to a 
set of questions frequently 
asked by policymakers, state 
legislatures, local boards of 
education, and other 
governing bodies about 
secondary and postsecondary 
CTE. 
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Furthermore, note that each of these purposes apply equally to the different operational levels — 
state, school district or college, and program. 

The Many Different Ways to Conduct ROI for CTE 

The list below li nk s each of the three operational levels — state, school district or college, and 
program to the three purposes — rational decision-making, informed choice, and validating 
strategic planning. In theory, ROI for CTE may be undertaken in nine different ways, moving 
from the most simple to the most complex: 

1 . The most simple way is when the operation level is a program and the purpose is rational 
decision-making. 

2. Less simple is when the operation level is still a program, but the purpose is now 
informed choice. 

3. Even less simple is when the operation level is still a program, but the purpose now 
becomes an effort to validate a strategic plan. 

4. The intermediate way is when the operational level is at the sub-state level and the 
purpose is rational decision-making. 

5. The more intermediate way is when the operational level is at the sub-state level but the 
purpose now is making informed choices. 

6. The most intermediate way is when the operational level is at the sub-state level and the 
purpose becomes validating a strategic plan. 

7. More complex is when the operation level is at the state level but the purpose is rational 
decision-making. 

8. Even more complex is when the operation level is at the state level, but the purpose now 
becomes making an informed choice. 

9. The most complex involves operating at the state level with the purpose of validating the 
strategic plan. 

In reviewing the above list, it is clear that from among the nine, a lesser number actually are 
applicable in practice. For example, 1,5,6 and 9 are the ways in which the three different 
operational levels — state, school district or college, and program — typically apply ROI. 

At the program level, rational decision-making is relevant because operationally SEAs and LEAs 
need to know if a particular program is internally efficient (in terms of cost) and externally 
effective (in terms of meeting the desired goal). On the other hand, if the focus is a single 
program, then 2 and 3 are not relevant. At the sub-state level (school district or community 
college), 5 and 6 are applicable because here the thrust is either to make an internally efficient 
and externally effective choice or to conduct an ROI study to validate a strategic plan. Sub-state 
levels are less interested in judging the internal efficiency and external effectiveness of a single 
program, although at times this may be required if a program is sufficiently large. Finally, at the 
state level, the relevant way in which ROI for CTE can be operationalized is represented by 9. 

By so doing, 7 and 8 are also covered in terms of internal efficiency and external effectiveness. 
Appendix A provides examples of each of the above various ways in which the different levels — 
state, school district or college, and program — operationalize ROI by engaging in a process that 
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either leads to making a rational decision, making an informed choice, or validating a strategic 
plan. 

The first section of this report discussed why ROI for CTE has been difficult. There it was 
argued, broadly speaking, that CTE faces budgetary, cultural, and institutional constraints when 
it tries to conduct ROI for CTE. The Building Blocks for ROI Studies laid out the building blocks 
for how to conduct ROI for CTE, placed ROI within the broader framework of program 
evaluation, and indicated the importance of why internal efficiency needs to be separated from 
external efficiency. Connecting CTE to ROI Measures discussed the reasons, purposes, and ways 
in which ROI for CTE can be conducted. Setting the Stage for Measuring ROI for CTE, the next 
section of this report, takes information from each of these previous sections to set the stage for 
measuring ROI for CTE using a common protocol. 
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Setting the Stage for Measuring ROI for CTE 


Stone and his colleagues (Stone et al., 2010) reviewed the education ROI literature and identified 
three strands of discussion regarding the various approaches to and methods of measuring ROI. 
The first focuses on the theoretical foundations for developing ROI approaches and methods. 
Much of this discussion is complex and requires an advanced knowledge of the mathematical 
application of concepts in finance and economics. A good general review of these theoretical 
foundations can be found in Pscharapolous (2006), which also provides a policy context for ROI 
in education and presents evidence for why education has value. 

Second, as highlighted in the Stone et al. report, many studies have produced ROI for education 
measures, some more specific than others. In general, these studies have applied many of the 
ingredients, tools, and building blocks described in the first section of this paper, The Building 
Blocks for ROI Studies. Some of these studies are highlighted in Appendix A. A key point is that 
education ROI analyses “have been applied using different goals, techniques, settings, and data” 
(Stone et al., 2010, p. 8). Consequently, we recommend that CTE ROI studies follow a common 
protocol. Following the recommendations for all of education made by Hummel-Rossi and 
Ashdown (2002), a framework for this common protocol for CTE ROI analyses is provided at 
the end of this section. 

A third strand in the education ROI literature identifies the different approaches that have been 
used to conduct ROI. The remainder of this section focuses on the three most common 
approaches employed in ROI in education studies. 

ROI Can Be Conducted Using Different Techniques 

Thus far, this report has discussed the challenges faced when conducting ROI for CTE and 
described reasons for undertaking ROI for CTE. No matter the challenges or the reasons for 
doing so, the building blocks for conducting ROI for CTE are the same, whichever of the 
following techniques are used. 

Education ROI analyses are generally conducted using three different techniques. They are: 

• The common framework technique must meet the following four preconditions: It must 
include (a) well-developed, integrated conceptual frameworks, (b) advanced institutional 
research and evaluation expertise, (c) highly connected data systems, and (d) sound data 
administration and superior management knowledge and oversight. This technique 
requires, at a minimum, explicit linking of education and workforce databases to measure 
the impact of a particular investment on both direct and indirect beneficiaries. 

• The social benefits technique measures the total benefits that accrue from initiating a 
specific programmatic action (e.g., reducing the number of CTE dropouts). The 
programmatic action has both direct and indirect impacts. The direct impact is quantified 
as that which immediately benefits the program’s target population. The indirect impact 
is generally quantified as a measure of the gain to the community. For example, an 
indirect impact might be: By moving “X” number of CTE students from dropout to 
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graduation status, a “Y” increase in tax collections would result. Matching these direct 
and indirect benefits against program costs provides a social benefit-cost ratio for the 
action. 

• The case study technique identifies the factors that influence the success of selected CTE 
programs and whether such programs are achieving a reasonable ROI compared to 
similar programs. A program may be a school, a specific occupational program within a 
school, a particular pedagogy, or an administrative structure that leads to improved 
student performance. 

Recently, the NRCCTE produced a CTE ROI study that provided a primer for conducting ROI 
under the four preconditions required by the common framework technique (Hollenbeck, 2011). 
The Hollenbeck study found that participants in CTE programs reaped substantial returns — 
positive earnings — with almost nil or negative costs associated with secondary CTE. At the 
postsecondary level, any associated participation costs (e.g., tuition, foregone earnings) were 
more than outweighed, even over the short term, by the economic payoffs of participating in 
CTE. The common framework technique used by Hollenbeck has been applied to CTE only 
infrequently, however, because most states and districts find it difficult to meet all four of the 
preconditions. Therefore, the field needs to find less stringent alternatives without sacrificing 
rigor when the common framework technique is too difficult to conduct. 

A typical alternative approach is to cull data on benefits and costs from a variety of different 
studies (such as those presented in Appendix A) and apply the social benefits technique. What 
distinguishes the social benefits technique from the case study technique is a matter of scale: The 
social benefits technique is generally used when ROI calculations involve both direct and 
indirect effects, and the latter usually outnumbers the fonner. The case study technique focuses 
more on direct impacts. 

Consider this example. An Alliance for Excellent Education study (AEE, 2010) calculated that 
about $260,000 per dropout would be saved over a student’s lifetime if the current dropout rate 
(30 out of 100 students) were reduced to zero. This study also noted that graduating from high 
school generates approximately $10,000 in additional annual income. The case study technique 
might focus on this last number. About 1.3 million high school students drop out every year. 
AEE estimated that the lifetime economic gain reaped by eliminating high school dropout would 
be about $335 billion, a number that would be provided when applying the social benefits 
technique. The NRCCTE has been conducting detailed analyses of NCES Sample Survey data 
and developed a new typology of CTE credit-taking (Kotamraju, Aliaga, & Dickinson, 2011). 
Using this typology, the authors estimated that nearly half of all high school graduates take at 
least three or more CTE credits. NRCCTE researchers have also established that high school 
students completing three or more CTE Carnegie credits are less likely to drop out than those 
taking between zero and one CTE credits (Aliaga, Stone, Kotamraju, & Dickinson, 2011). We 
can thus argue that approximately $168 billion of the lifetime gain from reducing the dropout 
rate to zero can be attributed to intensive CTE courses. Such ROI calculations were made 
possible by using different studies that each applied unique assumptions and specific 
methodologies. In such conditions, care must be taken to state and describe these differences 
clearly, addressing any inconsistencies and how they have been handled. 
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A Protocol for Guiding Future ROI for CTE 

In reviewing ROI for education, Hummel-Rossi and Ashdown (2002) suggested the use of a 
common protocol to guide future analysis, arguing that the lack of a general and uniform 
protocol has limited the field’s ability to evaluate ROI in education studies. More recently, 
however, studies have applied a single protocol to judge different ROI in education approaches 
and methods (Ross, Barkaoui, & Scott, 2007; Yeh, 2010). Hummel-Rossi and Ashdown (2002), 
for example, developed a common protocol for educational ROI that has nine components: (a) 
perspective, (b) cost analysis, (c) comparators, (d) program effects, (e) outcome measures, (f) 
distributional consequences, (g) time-effect analysis, (h) sensitivity analysis, and (i) decision 
rule. In addition, they suggested that a full-fledged ROI should be developed and used as a 
reference case for future, similar educational ROI studies. Hummel-Rossi and Ashdown (2002) 
then took each component and provided an educational context in which the application for the 
protocol would be optimal. In the sections that follow, the context in which ROI for CTE can be 
applied is discussed for each of the nine components of the Hummel-Rossi and Ashdown 
protocol. 14 

Perspective refers to the “goals of the evaluation that are clearly articulated and for which there 
is consensus” (Hummel-Rossi & Ashdown, 2002, p. 20). Identifying appropriate goals for ROI 
for CTE has been problematic because CTE involves many definitional, technical, and policy 
issues (Kotamraju, 2012) that make development of a common protocol for ROI for CTE more 
difficult. As this report has highlighted, the reasons for evaluating ROI for CTE — such as 
accountability requirements, program improvement, and marketing — at least provide the context 
around which the goals for ROI for CTE can be developed. Additionally, knowing the 
operational constraints at the state, sub-system, or project level can help delineate the contours of 
an ROI study. 

A major missing ingredient for CTE is the need for a common perspective. That common 
perspective can be found in recent non-regulatory guidance provided by OVAE with regard to 
rigorous programs of study (RPOS). RPOS are those programs of study (POS) that adhere to a 
basic framework that includes 10 components (see Appendix B). The 10 RPOS components 
expand and refine the original four elements (of which only three are actually required) of POS 
laid out in the Perkins IV legislation, which also mandated that all states must implement at least 
one POS in order to receive funds. National adoption of these 10 components would provide a 
common perspective when ROI for CTE is undertaken at the state, sub-system, or program 
levels. However, how to convert these components into actions and measures remains a 
challenge. Moreover, the question of whether all or some of the 10 components need addressing 
is yet to be detennined. 

In The Building Blocks for ROI Studies, five items — (a) the opportunity cost, (b) the time 
horizon, (c) the discount rate, (d) monetization, and (e) externalities — were identified as building 
blocks in any ROI study. These five items are connected to five of the nine components in the 


14 The protocol developed by Hummel-Rossi and Ashdown (2002) was specifically for measuring cost effectiveness. 
The protocol would not differ very much if it were generalized to all of ROI, as defined in this report. 
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common protocol — cost analysis, distributional consequences, outcome measures, time-effect 
analysis, and decision rule. Cost analysis requires identifying all costs, including opportunity 
costs. Opportunity costs are generally defined as costing out all alternative actions and decisions 
that are no longer possible; when the ROI is undertaken, it generates specific and particular 
associated costs, some of which maybe implicit. Those costs that are implicit in nature need to be 
monetized using particular decision rules like the discount rate. Distributional consequences 
imply taking into account the different externalities (positive and negative) that result from 
conducting the ROI. The distributional consequences also give rise to indirect outcomes, over 
and above the direct impact, that result from conducting the ROI. Which outcomes get included 
in the process depends on the time horizon; the time-effect analysis varies depending on the time 
horizon. Once again, the choice of a discount rate becomes crucial. Taken as a whole, one could 
argue that the five protocol components, which are subjective in nature, get translated objectively 
into the five items (building blocks), so that some quantification within the ROI can be attained 
and a (numerical) decision rule can be applied. 

Several additional points, however, need to be made. First, Perkins funds that are allocated 
toward and spent on CTE-related activities at the program, sub-system, and state levels can be 
used as the overt (explicit) costs in a cost analysis. States include such infonnation (in the 
aggregate) as part of their CAR submission. Nevertheless, “hidden” costs need to be 
acknowledged because CTE (or, for that matter, any other programmatic effort) is typically 
embedded within the larger educational cost structures at the state, sub-system, and program 
levels (Boser, 2011). Federal funding is actually a very small portion of the overall investment 
any state makes in CTE. The example of one state — Minnesota — illustrates this fact. 

Perkins provides approximately $20 million annually to Minnesota to support CTE programs at 
the secondary and postsecondary levels. This is a relatively small investment when compared to 
education spending as a whole for the state. The state’s K-12 annual education budget is 
approximately $15.1 billion, with the higher education budget approximately $3.2 billion (Smith 
& Kotamraju, 2008). The percentage of the federal investment in CTE is therefore only a 
fraction of overall state spending on education. Although the Minnesota investment in CTE is 
much smaller than in other states (e.g., Oklahoma, Pennsylvania, Kentucky), nevertheless, the 
federal investment does much to provide the catalyst for how state and local funds are allocated 
towards CTE. 

Second, each particular costing approach entails distributional consequences. If only Perkins 
funds are used in the costing method, then the ROI for CTE analysis actually measures the 
efficiency of Perkins operations at different levels. In other words, the analysis is restricted to 
measuring internal efficiency and leads toward an answer to the first question: Is the federal 
investment of Perkins dollars paying off? However, a larger question involves the measurable 
impact of CTE beyond the level at which Perkins funds are applied. In other words, the question 
focuses on external effectiveness — the worthiness of CTE to overall education and workforce 
development efforts at the state, sub-system, and program levels. One way to address this 
question is to build a set of interactions among and between the 10 POS components and to cost 
them out. These interactions are shown in Appendix C, which offers one way to identify and 
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determine what the appropriate distributional weights might be among myriad interactions and 
how they are included in the external effectiveness measure. 

Third, as indicated in the introduction to this report, identifying standardized outcomes for use in 
a CTE analysis has proven difficult. The Perkins accountability indicators may serve as outcome 
measures. Table 3 displays the core accountability indicators that are required under Perkins IV. 

Table 3 

Perkins IV Core Accountability Indicators 


Secondary Level 

151 - Academic Attainment in Reading/Language Arts 

152 - Academic Attainment in Mathematics 
2S1 - Technical Skill Attainment 

3S1 - Secondary School Completion 
4S 1 - Student Graduation Rate 

5S1 - Placement in Postsecondary, Employment, or 
Military 

65 1 - Nontraditional Participation 

652 - Nontraditional Completion 


Postsecondary Level 


1P1 - Technical Skill Attainment 
2P 1 - Credential, Certificate or Diploma 
3P1 - Student Retention or Transfer 

4P 1 - Student Placement in 
Employment 

5P1 - Nontraditional Participation 
5P2 - Nontraditional Completion 


Note: Nontraditional in the CTE context refers to students enrolling in and completing programs that are 

nontraditional for their gender. Men in nursing or women in welding are considered prototypical examples of 
students enrolling and completing CTE programs that are nontraditional for their gender. 


LEAs collect and report CTE and other data to SEAs, who then take this data to calculate the 
numerical values for each of these indicators, which they then report these as part of their annual 
submission of the CAR. The current Perkins accountability system has separate indicators for 
secondary and postsecondary and, notably, the two sets of indicators cannot be connected to one 
another. This is because, for each set, measurement definitions are unique, data collection 
methods vary, and reporting procedures are different. The same holds true for indicators within 
each set. The need to connect the different secondary and postsecondary Perkins indicators has 
become crucial given the heightened interest in finding appropriate outcome measures for POS. 
Some have argued that the ultimate test for students successfully enrolling and completing a POS 
is finding and keeping employment that is closely related to the POS in which they enroll and 
complete. In ROI studies, successful placement in employment is seen as benefit, and therefore 
has the potential of increasing the internal efficiency and external effectiveness of the enterprise. 
Therefore, when conducting ROI for CTE, secondary and postsecondary indicators must be 
connected to one another, as well as the indicators within the secondary set and within the 
postsecondary set. 

As discussed below, the ideal data system for conducting ROI for CTE is a longitudinal data 
system in which a cohort of students enroll in, for example, a POS. Success is defined when 
students in the cohort complete their educational program and are placed in employment. The 
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Figure 4. Linking secondary and postsecondary Perkins accountability indicators: A schematic 
representation. 


advantage with using a cohort is that it reduces to a minimum any variations in measurement 
definitions, data collection methods, and reporting procedures, a basic pre-condition when using 
longitudinal data systems. Figure 4 uses current secondary and postsecondary Perkins 
accountability indicators to outline a process wherein the numerator in one indicator is the 
denominator in the subsequent indicator, and the denominator in the same indicator is the 
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numerator in the previous indicator. In the context of this report, one could view the numerator 
as benefits and the denominator as costs, adjusting each as students within the cohort progress 
from enrollment in high school to eventually being placed in employment. Cohort analyses can 
also be adjusted to include those students whose progression within the education system is not 
linear. The state of Minnesota employs a cohort analysis when it reports its Perkins 
accountability information at the postsecondary level (Minnesota State Colleges and Universities 
and the Minnesota Department of Education, 2011). Recently, keeping the goal of a well- 
designed accountability system in mind, CTE has begun exploring ways to remake the Perkins 
accountability system through the State Perkins Accountability Congress (SPAC). 15 More 
generally, as states begin developing and implementing longitudinal data systems across the 
nation, moving toward a cohort-based analysis should become the preferred way for conducting 
ROI for CTE. 

Fourth, in any ROI analysis, careful consideration must be given to reconciling the nature of 
multi-year programming within education and the annual budgeting processes that states, sub- 
systems, and programs have to put in place. CTE is no different. The Perkins legislation usually 
runs over a period of six years, but funding is generally restricted to annual allocations. Benefits 
and costs must therefore be annualized, adjusted for inflation, and discounted accordingly. 

Fifth, the decision rule applied within any ROI analysis is typically a number (e.g., cost-benefit 
ratio, net present value, or internal rate of return). Hummel-Rossi and Ashdown (2002) suggested 
that such numbers are the starting point in any ROI analysis. Other decision-making tools, 
contextual in nature, should be given equal, if not more, weight. As FIummel-Rossi and 
Ashdown (2002) stated, “Societal values, such as equal opportunity for learning and 
developing. . . may not be consistent with economic efficiency, and, consequently, these values 
must be weighed against [the purely numerical information]” (p. 22). CTE would do well to 
strike this balance, particularly when it has been repeatedly suggested that CTE has ignored the 
data needed to determine internal efficiency and external effectiveness (Duncan, 2011) 

If CTE is to include the three remaining components of the FIummel-Rossi and Ashdown (2002) 
protocol — program effects (use of rigorous statistical techniques), comparators (finding 
reasonable alternatives for comparing), and sensitivity analysis (checking the robustness of the 
analysis) — then the only appropriate applicable technique is the common framework technique 
(Flollenbeck, 2011). For CTE, technical, analytical, and budgetary constraints make this 
particular technique difficult. Nevertheless, an analyst intending to measure the ROI for CTE can 
employ one or more of the following approaches to develop the kind of studies that have been 
outlined in Appendix A. 

The place for the CTE field to start would be to use the case study technique. Such studies would 
be mostly done at the classroom or program level. This would permit the analyst to develop the 
necessary basic skills, leam the different terminology, and apply the simple tools that have been 
discussed in this report. It would also enable the analyst to identify the limits of the case study 
technique, particularly when it comes to measuring indirect impacts. To continue perfecting his 


15 Further detail regarding this effort can be found on the Perkins Collaborative Resource Network website: 
http://cte.ed.gov/index.cfm . 
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or her skills, the analyst can apply the case study technique to several situations, be these at the 
classroom or the program level. Well-crafted case studies can prove useful for making the case 
for program improvement. If the social benefits technique is to be used, usually at the sub-state 
or state levels, the focus of the ROI study is social marketing, in which in addition to the four 
“Ps” of commercial marketing — product, price, place, and promotion, additional “Ps” — publics, 
partnerships, policies, and purse strings, can be included as well (Weinreich, 2006). The main 
difference between the case study technique and the social benefits technique is the scale at 
which the ROI is conducted. The social benefits technique is typically used on projects and 
programs that have impacts beyond themselves (hence the term social). Also, by applying the 
social benefits technique to various broad programs and projects, the analyst may develop a 
better sense and knowledge of the boundaries and limitations of doing ROI studies. This is 
extremely important because the use of the social benefits technique in conducting ROI studies 
has often resulted in grossly exaggerated claims (see McHenry, Sanderson, & Siegfried, 2011). 

However, if an SLDS is in place, applying the common framework technique becomes easier. 
Moreover, an SLDS would make implementing the common protocol described in this report 
possible and permit a move toward standardization. It would allow for having a common 
perspective on goals (outcome measures) such as the four areas of focus within the U.S. 
Department of Education: (a) strengthening POS, (b) improving data and accountability, (c) 
increasing teacher effectiveness, and (d) turning around low-perfonning schools. Costs and 
benefits can be derived from the same data platfonn. Program effects can be based on a 
consistent set of comparators, with distributional consequences assessed uniformly, sensitivity 
analyses based on a common set of parameters, and decision rules applied using appropriate and 
well-defined rules. Finally, the time frame for conducting an ROI can be made to be the same no 
matter which strategy and what program is being assessed. In short, the common framework 
technique allows for all three purposes for ROI for CTE — program improvement, accountability 
and evaluation, and marketing — to be achieved simultaneously. 

The CTE community needs to find ways to make the use of the common framework technique 
more widespread if it is to gather the hard evidence to challenge popular misperceptions the 
broader education and workforce communities might have regarding the internal efficiency and 
external effectiveness of CTE. It is noteworthy that the CTE community has recently been asked 
to participate, both at the state and national levels, in the growing discussion regarding 
developing, maintaining, and sustaining SLDS. 16 CTE has begun to make the first movements 
toward implementing ROI analyses that use the common framework technique by involving 
itself in the development of comprehensive SLDS. Using scientifically based research methods, a 
standardized data system such as SLDS, and the ROI building blocks, tools, and techniques 
described in this report, the CTE community could systematically develop several CTE ROI 
analyses. 


16 Recently, a group of CTE accountability specialists attended the SLDS P-20W Best Practice Conference in 
Arlington, VA (November 15-16, 201 1). There was much interest in what CTE could contribute toward building a 
strong SLDS, both from an education and a workforce development perspective. Resulting from this participation 
was the formation of a CTE SLDS workgroup (comprised of an equal number of CTE and non-CTE accountability 
specialists). The group has the potential to address CTE data issues and influence CTE evaluation and research via 
SLDS (personal communication, Sharon Enright, Associate Director, CTE Performance and Accountability, Office 
of Career and Technical Education, Ohio Department of Education). 
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In the meantime, using the case study approach, SEAs and LEAs would do well to look inwards 
at the myriad CTE programs that have been, or currently are being, implemented across the 
country. There is a wealth of infonnation present in those programs that can provide the 
ingredients necessary for conducting ROI for CTE. Aggregating case studies within a state 
community college system, for example, might offer a lower cost approach to a useful ROI for 
CTE if system cannot afford or have the necessary data to follow the common framework 
technique. It also would limit the use of the social benefits technique, which can be somewhat 
misleading approach because it can exaggerate benefits and underestimates costs by ascribing 
more to the results that what might actually be the case. This report recommends that the CTE 
community focus on the case study approach in the shorter term, and as it continues to push for 
the inclusion of CTE data and information within the SLDS framework in the longer term, which 
would make conducting ROI for CTE using the common framework technique that much easier. 
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Appendix A 

Examples of ROI Studies 


Using materials obtained through standard literature review practices, we created individual 
summaries of major findings and key infonnation relevant to ROI analysis for CTE. Each 
summary was then categorized as to its type of study and type of ROI analysis, using definitions 
and parameters for each as outlined in this report. Studies in which the type of analysis 
delineations were inconclusive or subjective were initially left undefined and then re-categorized 
so as to best represent the majority of the more objectively defined studies previously identified. 

In the summaries that follow, these abbreviations are used: CF = Common Framework, SB = 
Social Benefit, and CS = Case Study. 


State Studies 


Alabama 

(SB) Smith, D. W. (2006). A return on investment study for an engineering company in 

Huntsville, Alabama using a community college web-based training program. (Doctoral 
dissertation, Mississippi State University). Retrieved from ABI/INFORM Global. 
(Publication No. AAT 3211246). 

http://proqucst.um i.com/pqdlink?did= 1 1267743 1 1 &Fmt=7&clicntld=47297&RQT=309&VNam 
e=PQD 

Dissertation, Project Level 

This ROI analysis focused on the expansion of a joint private engineering company and 
community college web-based training program following an initial pilot study. Under the 
expansion of the program, all employees of the private company were afforded the opportunity 
to participate in the web-based training program. ROI per training participant was 1,755%, 
$17.55 per investment dollar, an increase from 121% in the original pilot study. ROI did not 
correlate with gender, age, ethnicity, level of education, or number of courses taken by the 
training participants. Positive financial results were attributed to the expansion of the program, 
participants were satisfied with the flexibility, accessibility, and convenience of online training. 

Alaska 

(CS) Phipps, R. (2008). Making Alaska more competitive by preparing citizens for college and 
career. Washington, DC: Institute for Higher Education Policy. 

http://www.eed.alaska.gov/edsummit/pdf/MakingAKMoreCompetitive.pdf 

Report Executive Summary, State Level 
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The Alaska Commission on Postsecondary Education, in partnership with the National College 
Access Network, surveyed the postsecondary access and success climate within the state of 
Alaska. As an underperforming educational state, the survey revealed that a college-going 
culture does not exist within Alaska and a causal link between Alaska’s workforce needs and 
postsecondary education has not been publicized. The report offered five recommendations to 
improve educational performance in Alaska. These recommendations consisted of developing 
strategies that will create a statewide college-going culture, establishing a partnership 
environment among postsecondary, K-12, business, and community groups, creating a peer 
mentoring program to enhance college access, building Alaska Advantage programs to increase 
postsecondary opportunity awareness, and requesting cabinet-level attention and legislation to 
the postsecondary education access for Alaskans. 

California 

(SB) Belfield, C., & Levin, H. (2009). The return on investment for improving California ’s high 
school graduation rate. Santa Barbara, CA: California Dropout Research Project. 

http://schooliinance.org/resource center/research/CADropout-Research2.pdf 

Report Abstract, State Level 

Educational investments and interventions are evaluated within this report as either evidence- 
based, high-efficacy interventions or promising but unsubstantiated educational interventions. 
Average applicable implementation costs and a per additional high school graduate figure were 
calculated for both types of educational investment before being compared to the economic and 
social benefits to taxpayers and the state of California. Specifically, California will obtain fiscal 
benefits of $53,600 per new high school graduate and will additionally accrue social benefits of 
up to $391,900 per student; the federal government will receive an additional $1 15,300. State 
and local funding initiatives were found to have a return of $53,600 while those initiatives 
funded through a combination of local, state, and federal funding were found to generate 
$189,000 per student. Of the educational interventions surveyed, most were justified on a cost- 
benefit basis, in that the average California K-12 education costs the public $170,420 — a figure 
less than the graduate return of $189,000. 

(SB) Gonzalez, R. G., & Kohli, A. (n.d.). Getting a return on investment: The California 

DREAM Act. Berkeley, CA: The Chief Justice Earl Warren Institute on Race, Ethnicity & 
Diversity. 

http://www.law.berkeley.edu/files/CA DREAM Act Return on invstmnt Final.pdf 
Article, Project Level 

This article provides return on investment information relative to the institution of the California 
DREAM Act (SB 1301) and Assembly Bill 2083 that would provide undocumented students the 
opportunity to compete for state financial aid for postsecondary education. Using information 
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from a recent RAND study, $ 1 5 million per year in net tax revenue could be generated by the 
estimated 1620 undocumented students currently enrolled in California’s colleges, post- 
graduation. Bureau of Labor Statistics estimates found that the average bachelor’s degree 
recipient earned $962 per week compared to $419 per week for non-high school diploma 
recipients. Another RAND study stated that a Mexican immigrant woman with a college degree 
will pay $5,300 more in taxes and cost $3,900 less in social support compared to a dropout. 
Related studies revealed that postsecondary enrollment of undocumented students increased ten- 
fold when eligibility was established for financial aid in Texas and that $2.5 million would be 
generated in Massachusetts over three years if undocumented students received in-state tuition. 

(CS) Stange, K. (2005). The economic impact of the F oothill-De Anza community college district 
and its students. Berkeley, CA: University of California, Berkeley. 

http://www-personal.umich.edu/~kstange/policv/FHDAEconImpact.pdf 

Report Summary, Project Level 

The Foothill-De An / a community college district consists of two two-year colleges within the 
Silicon Valley that collectively educate more than 60,000 students while employing 3,000 
faculty and staff members. The ROI analysis revealed that Foothill-De An / a is directly 
accountable for $277 million of local economy expenditures, with an additional $186 million in 
indirect spending and a further $353 million in annual alumni earnings attributed to increases in 
productivity. A $3.67 local economic impact return on investment was calculated per property 
tax dollar, with $1.95 in direct district spending and $1.88 in indirect spending and additional 
alumni earnings. 

(SB) Robinson, M. H., & Christophersen, K. A. (2008). The economic contribution of the Los 
Angeles community college district. Moscow, ID: Economic Modeling Specialists, Inc. 

http://research.laccd.edu/eir/ 

http://research.laccd.edu/eir/LACCD ES Final%20v3.pdf 
Executive Summary, Project Level 

This study was based upon multiple economic growth and investment analyses pertaining to the 
Los Angeles Community College District (LACCD) relative to returns on investment for 
taxpayers, the state of California, and students themselves. Specific findings pertaining to 
students included a 24% annual return on investments, and a $7.60 return per dollar investment 
in LACCD in cumulative higher future income. The LACCD was found to provide a real money 
return of 10% for annual taxpayer investments, whereas the State of California has annual 
projected savings of $38.5 million for LACCD students within the workforce. Regionally, the 
LACCD accounted for 1.8% of Los Angeles County annual income — an annual impact of 
approximately $9.8 billion. 
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(SB) Heller, D.E. (2005). Public subsidies for higher education in California: An exploratory 
analysis of who pays and who benefits. Educational Policy, 19(2), 349-370. doi: 
10.1177/0895904804273542 

http://epx.sagepub.eom/content/19/2/349 

Abstract, State Level 

This exploratory study analyzed the distribution of postsecondary education benefits obtained 
within the state of California on a racial group basis. Asian American students were found to 
receive proportionally higher amounts of state funding compared to Latino students despite their 
widely divergent total student populations. This funding and corresponding benefit difference 
was attributed to observed racial group differences in two- and four-year college attendance rates 
along with associated instructional costs and public subsidies. Policy implications for 
postsecondary education funding and participation rates were discussed in addition to areas of 
future research pertinent to this area. 

Connecticut 

(SB) Whipple, S. (2008, November 2). State getting return on community college investment. 
The New Britain Herald. Retrieved from 

http://www.newbritainherald.eom/artieles/2008/l l/02/news/doc490e6e4e0197d97334135 
4.prt. 

http://www.newbritainherald.com/articles/2008/ll/02/news/doc490e6e4e0197d973341354.prt 
Article, State Level 

This article details the findings of a 2008 economic impact study of Connecticut’s community 
colleges, focusing upon return on investment from multiple analytical perspectives. Key findings 
of this study included $5 billion in annual economic input from two-year institutions and their 
students, amounting to 2.3% of total state annual income and a $16.40 return per tax dollar 
investment in cumulative student lifetime earnings. With enrollments at two-year Connecticut 
institutions at an all-time high, student returns on investment were also detailed, with a 
calculated $23 1 annual return per credit hour completed; lifetime earnings were also shown to 
have a $8. 10 return per dollar invested amounting to an additional cumulative $600,000. 

Florida 

(CF) Florida Department of Education. (2005). Return on investment/school efficiency measure 
(linking learning and costs). Tallahassee, FL: Florida Department of Education. 

http://roi.fldoe.org/index.cfm?CFID=3236386&CFTOKEN=574bed5cd7fe2fcl-A52E639E- 
5056-8C3F-1 66427389 1A9CFB1 
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Database, School (Project), District, and State Level 

This Florida Department of Education database provides school, district, and state level 
measurements of ROI and school efficiency in an effort to improve the efficiency of delivering 
quality services within the state’s public school system. Comprehensive student, staff, and 
finance data systems, maintained by the state’s department of education, comprise the primary 
information sources for the database. The intent of this website was to enable users to be able to 
evaluate relative ROI based upon allocation of fiduciary resources and academic student 
performance on multiple levels of analysis. 

(CS) Florida Department of Corrections, Bureau of Research and Data Analysis. (1999). Return 
on investment for correctional education in Florida. Tallahassee, FL: Florida Department 
of Corrections. 

http://www.dc.state.fl.us/pub/taxwatch/index.html 
Executive Summary, Sub-System Level 

This ROI analysis focused upon a cost-consequences analysis of the per-dollar ROI of 
correctional education within the state of Florida. The Florida Department of Labor and 
Employment Security undertook this study in order to detennine whether a “coarse-grain 
analysis” could be applied to Job Partnership Training Act programs to measure the translation 
of educational inputs into earnings. First and second year returns on investment by program area 
were provided with a general per-dollar ROI figure of $ 1 .66 for the overall study; all subgroups 
of correctional education completers were found to have positive returns on investment. After the 
second year, ROI figures increased to an average of $3.20, with non-special education academic 
completers having the highest return at $3.53. It is noted that non-completers of the correctional 
education program were not factored into the ROI analysis. 

Georgia 

(SB) Tripp Umbach Consultants. (2008). Expanding medical education in Georgia: Roadmap 
for Medical College of Georgia School of Medicine and statewide partners - Final 
report. Pittsburgh, PA: Tripp Umbach Consultants. 

http://www.usg.edu/health workforce center/documents/Expanding Medical Ed in GA Final 
Report.pdf 

Report Abstract, State Level 

Population growth and a lack of investment in medical education within the state of Georgia 
have resulted in a severe deficit of medical doctors that will approach crisis levels by 2020. As a 
result, the Medical College of Georgia School of Medicine must expand rapidly in multiple 
locations and in partnership with the University of Georgia. An independent consulting firm 
(Tripp Umbach) developed a plan that would generate an addition $ 1 .6 billion annually while 


36 



providing an additional 10,000 jobs within the state of Georgia. The specific per-dollar ROI 
figure cited that for every $1.00 invested by the State of Georgia in medical education, $2.54 will 
be generated in state tax revenue. 

Idaho 

(SB) Midgley, J. S. (2011). Return on postsecondary education investment: An analysis of 
professional and technical education degrees in Idaho. (Unpublished doctoral 
dissertation, University of Idaho). 

Dissertation, State Level 

This dissertation approached the evaluation of professional technical postsecondary education in 
Idaho using a quantitative approach based upon human capital theory. ROI calculations were 
determined by analyzing entry career earnings relative to the costs of earning applied associates 
and baccalaureate degrees, respectively. ROI calculations were based on data obtained from 
postsecondary institutions, the Idaho Department of Labor, and the Division of Professional 
Technical Education of Idaho. Key variables within subsequent analyses included ROI relative to 
field of study, entry wages, and number of academic credits earned. Although baccalaureate 
degrees were categorically associated with higher wages, ROI calculations, which included 
tuition costs for applied associated degrees, were found to exceed those of baccalaureate degrees. 
Although wages and ROI were found to be highly correlated with specific occupational fields, 
when factoring in educational costs, postsecondary GTE associate’s degrees provided greater 
financial benefits compared to bachelor’s degrees on an ROI basis within this cohort. 

Indiana 

(CF) Hollenbeck, K. (2009). Return on investment analysis of a selected set of workforce system 
programs in Indiana. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. 

http://www.indianachainber.com/index.p h p/workforcesystems 

Report Executive Summary, State Level 

The Indiana Chamber of Commerce Foundation contracted with the Upjohn Institute to use 
Indiana Workforce Intelligence System (IWIS) data to conduct return on investment analyses of 
various workforce programs. The programs included within the analysis consisted of Workforce 
Investment Act (WIA) for Adults, Dislocated Workers, and Youth, Trade Adjustment Assistance 
(TAA), Sub-baccalaureate Postsecondary Education, and Work One. Fiscal year 2006 was used 
as the basis for all estimates with IWIS data from 2003-2008 providing estimate information. 
State and federal governments were found to benefit only over a working lifetime from WIA 
Dislocated Workers (1.50%), TAA (5.01%), and Postsecondary (1.82%). Individual rates of 
return were much more significant with WIA Adults (16.32%), WIA Dislocated Workers 
(2.64%), WIA Youth (13.27%), and Postsecondary (19.87%) all having positive returns. Benefits 
to society were also calculated, with primarily positive rates of return on investments. 
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(CS) Indiana Chamber (2009). Indiana ’s best buys: An in-depth look at Hoosier high schools. 
Indianapolis, IN: Indiana Chamber. 

http://www.indianachamber.com/media/pdf/IndianaBestBuy092.pdf 

Annual Report Purpose, State Level 

This annual report summarizes Indiana high schools that have excelled academically based upon 
SAT participation and scores, AP participation and scores, ISTEP+ and ISTEP+ pass rates, and 
adjusted graduation rates aggregated into a Quality Index (QI) compared to each school’s total 
revenues per pupil. A statewide public high school median was assessed at a QI of .0719 on 
revenues of $10,179 and an at-risk student population of 31.2%; specific findings were then 
reported for individual schools throughout the state with those exceeding the median QI of .0719 
on revenues at or below $10,179 being named best buys. This annual report is currently available 
from 1999-2009. 

Kansas 

(CS) Barton County Community College (2003). The economic impact of Barton County 
Community College on its Service Area, 2001-2002. Overland Park, KS: Office of 
Institutional Research, Johnson County Community College. 

http://bartonccc.net/ir/onhnereports/economic impact report bartoncounty.pdf 

Executive Summary, Project Level 

This report summarizes the findings of an economic impact study of Barton County Community 
College on its seven-county service area over 2001-2002. Direct expenditures by the institution 
as well as its constituents amounted to approximately $22 million; indirect economic benefits 
contributed an additional $19.8 million for a total service area economic impact of approximately 
$41 million. On tax revenues of $12.2 million, Barton County Community College had a per- 
dollar tangible economic contribution ROI of $3.42. 

Kentucky 

(SB) Watts, A. L. (2001). Education and the common good: Social benefits of higher education 
in Kentucky. Frankfort, KY: Kentucky Long-Term Policy Research Center. 

http:// eric . ed.gov/ERICW ebPortal/ detail?accno=ED4625 72 

Report Abstract, State Level 

This report detailed both the financial returns and social benefits that the state of Kentucky can 
expect from its investments in higher education, specifically those associated with or derived 
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from earning a degree from a four-year college. Twelve statistical models using datasets from 
semiannual surveys conducted by the University of Kentucky Survey Research Center were used 
in generating this analysis. Benefits of receiving a four-year degree included higher earnings and 
state and federal tax revenues, reduced welfare dependency, lower crime, healthier lifestyles, 
associated social gains, and various other benefits. Four-year degree recipients’ social benefits 
were valued in excess of $126,000 for men and $96,000 for women. Supporting datasets and 
figures are appended. 

(CS) Childress, M. (2008). Reducing obstacles will yield even higher academic returns to 

educational investments. Frankfort, KY: Kentucky Long-Term Policy Research Center. 

http://kltprc.info/policvnotes/pn0026 education funding.pdf 

Article, State Level 

Kentucky’s rate of return on investments in elementary and secondary education is among the 
best in the nation as assessed by the National Assessment of Educational Progress (NAEP) per 
$1,000 spent per student. Currently, Kentucky ranks 37 th in the nation in terms of NAEP 
proficiency; however, the Commonwealth ranks 25 th in the nation per $1,000 in per pupil 
spending at 3.6 NAEP proficiency points. Kentucky’s NAEP Proficiency Purchasing Power, 
which measures the cost effectiveness of administering elementary and secondary education, 
ranks eighth nationally at 118% of the predicted value, indicating that in spite of significant 
obstacles in providing quality education, higher academic returns are generated compared to 
other states. Such obstacles as poverty, parental education, rates of obesity and disability, and 
large size of rural populations place Kentucky fourth nationally on an Index of Obstacles. 

Maryland 

(CS) Maryland State Department of Education. (2008). Maty land adult education program quick 
facts FY ‘07. Baltimore, MD: Author. 

http ://www.msde.maryland . go v/N R/rdon I vres/2707E5 79-OD A0-442B-80 13- 
874CFAA74B00/17869/MD AEProgramQuickFacts07.pdf 

Annual Report, Project Level 

This quick facts sheet provides specifics of the Maryland Adult Education Program, Literacy 
Works, from the Maryland State Department of Education for the 2007 fiscal year. Specific 
returns on investment were reported earnings gains of $1,817 to $2,569 within six quarters 
following course completion and a $3 to $ 1 return on state educational investment. Additionally, 
the Maryland Adult Education program generated over $2.8 million in federal incentive funds for 
the fiscal year while meeting federal academic achievement performance targets for the past 
seven years. Specific funding sources included $9.2 million from Federal WIA Title II, $6.4 
million of state funding, and $3.1 million of local funding. 
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(CS) Maryland State Department of Education. (2005). Stepping up to the future: Adult literacy 
challenges at work, at home, and in the Maryland community. Findings and 
recommendations from the 2005 Superintendent’s Panel on Excellence in Adult 
Education. Baltimore, MD: Author. 

http ://www. eric . ed.gov/ERICW ebPortal/ detail?accno=ED492907 

Report Summary, State Level 

The Maryland State Department of Education detailed various returns on investment for its 
affiliated adult education programs in reporting findings and recommendations from the 2005 
Superintendent’s Panel on Excellence in Adult Education. Maryland data reported included 64% 
of unemployed adult education students finding a job within one fiscal year, and a recidivism 
reduction of 19% for inmate adult education participants, generating savings of $27,000 per year 
per student. An associated study that analyzed economic outcomes for Maryland adult education 
students found annual wage gains of between $1,817 to $2,579, an 18% to 25% gain for 
minimum wage workers; 120-180 hour program participants had 42% higher wage gains than 
others; and Adult Basic Education and English as a Second Language students with at least 120 
hours of instruction had earnings gains of 48% and 45%, respectively. ROI-specific information 
found a rate of return of $3. 15 per dollar invested, with 16,503 high school diplomas awarded 
and the average recipient earning $7,216 more per year — from these data, a 20-year return of 
$2.3 billion could be projected. 

Massachusetts 

(CS) Bluestone, B. (1993). UMASS/Boston: An economic impact analysis. Boston, MA: 
University of Massachusetts, Chancellor's Office. 

http ://www. eric . ed.gov/ERICW ebPortal/ dctail?accno=ED3 56733 

Report Executive Summary, Project Level 

The economic impact of the University of Massachusetts at Boston on the Commonwealth of 
Massachusetts is detailed within this report focusing upon student generated revenues, additional 
state income and sales tax generated from student incomes, and the income and tax revenues 
generated from external revenue sources into the university and state. This conservative analysis 
concluded that UMass/Boston has produced a significant return on investment from multiple 
perspectives. Key report findings included the 1991 student cohort generating $1.05 billion in 
additional in-state income flows. Although educating this student cohort will cost the state $34.1 
million, future income streams will yield state tax revenues of $53.5 million. The cumulative 
findings represent a $1.57 return on state per-dollar investments, and a related analysis found that 
eliminating state support for UMass/Boston would have negligible cost savings when lost 
revenues are taken into consideration. 
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Michigan 


(SB) Michigan Works! Association. (2008). Return on investment: A report on the public return 
on investment value of the Michigan Works! System. Lansing, MI: Author. 

http://michiganworks.org/media/ads/ROI 09 trifold brochure.pdf 

Pamphlet Calculations and Explanations, State Level 

The Michigan Works! Association is a workforce development system designed to advance the 
economy of the state of Michigan through public-private partnerships aligning industry needs 
with human resources. This pamphlet provides basic statistics and ROI findings across numerous 
programs at the federal and state levels, focusing upon their impacts within Michigan. Major 
findings include Workforce Investment Act returns on per-dollar investments of $ 1 .33 for WIA 
Adult, $1.37 for WIA Dislocated Worker, and $1.64 for Dislocated Worker with extension. The 
Jobs, Education, and Training program targeted toward welfare recipients was found to have a 
$4.13 return per dollar, whereas 31,000 people have entered training under the No Worker Left 
Behind initiative. Social returns on workforce development investment are also discussed 
through qualitative findings. 

(SB) Greene, J.P. (2000). The cost of remedial education: How much Michigan pays when 

students fail to learn basic skills. Estimates of the annual economic cost to businesses, 
colleges, and universities to counteract employees ’ and students ’ lack of basic reading, 
writing, and arithmetic skills. Midland, MI: Mackinac Center for Public Policy. 

http://www.eric.ed.gov/ERICWebPortal/detail7accnoM/D451288 

Report Abstract, State Level 

The purpose of this report was to assess the costs incurred by businesses and institutions of 
higher education within Michigan by having to provide remedial education. The methodology of 
this analysis focused on five different areas of expense: direct remedial education expenditures, 
direct costs to employers, costs for educating a successful high school graduate, using NAEP 
scores to determine the extent of deficits, and detennining an ROI for remedial education 
expenses. Annual loss calculations for the state of Michigan ranged from $311 million to $1.5 
billion with an average of $601 million across measures. The specific ROI calculation was based 
upon multiplying the number of dropouts in a given year (29,085) by the cost for educating a 
successful high school student ($6,552) by the average number of years a dropout misses (2.29), 
adding $89 million in higher education remedial expenditures and multiplying the result by 
1.03 30 (constant dollar with interest), yielding a $1.15 billion estimate of the cost to Michigan 
when high schools fail to teach students basic functional skills. 


Minnesota 
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(CS) Anton, P. A., & Behling, N. (2006). The economic impact of Minnesota state colleges and 
universities: Updated statewide estimates and local estimates for universities. Saint Paul, 
MN: Wilder Research. 

http://www.mnscu.edu/media/publications/pdf/statecollegereport 9-12-06.pdf 
Report Executive Summary, Sub-System Level 

The economic impact of colleges and universities within the state of Minnesota is detailed within 
this report, both on the state and local levels as well as with regard to postsecondary institutions’ 
capital expenditures. A statewide per-dollar ROI figure for postsecondary education was 
estimated to be $10.87, with the enhanced productivity of Minnesota workers accounting for 
$2.4 billion of a total statewide estimate of $3.5 billion. Capital expenditures by colleges and 
universities over the four years preceding the report generated $243 million in economic activity, 
whereas the 2006 higher education budget of $191.4 million will generate returns in economic 
activity of $430 million statewide. Local impacts of public universities were also provided in 
terms of the direct economic impact upon their communities. 

New Mexico 

(CS) Rommel, H. L. (2011). 2011 New Mexico adult basic education fact sheet. Santa Le, NM: 
New Mexico Higher Education Department. 

http://www.hed.state.nm.us/Adult Basic Education. aspx 
www.hed.state.nm.us/uploads/files/abe%20fact%20sheet%20 1 1 .pdf 

Annual Report, Project Level 

The New Mexico Higher Education Department Adult Basic Education Division consists of 28 
programs throughout the state designed to assist adult students in earning a GED high school 
diploma as well as career and college preparation. Although 23,000 individuals receive services 
through Adult Basic Education (ABE) programs, it is estimated that this represents only 5% of 
the target population; capacity issues within some areas are systemic, with extended waitlists for 
services. Annual ROI figures are provided, based upon annual state expenditures of $6,212,100. 
$20,913,360 in new income from job procurement and advancement was generated. $30,408,300 
was generated in enhanced earning potential from program graduates. An estimated $767,000 
was saved from students leaving public assistance. Cumulatively, these yielded an ROI of 
$45,876,560 for the ABE program at a cost of $267 per student. 
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New York 


(CS) Iatarola, P., & Fruchter, N. (2004). District effectiveness: A study of investment strategies 
in New York City public schools and districts. Educational Policy, 18, 491-5 12. doi: 
10.1177/0895904804265020 

http://epx.sagepub.eom/eontent/18/3/491 

Abstract, Sub-System Level 

This article focuses upon the impact of specific New York City Public School Districts upon 
their corresponding schools. Relative academic strength or weakness within the New York City 
school system correlated with differences in district educational goals, leadership and 
professional development, and teacher recruitment and retention. Policy implications are 
discussed in addition to previous studies and literature that support the major findings and 
conclusions arrived by this district-level analysis. 

Oklahoma 

(CS) Reynolds, E. J. (2005). Oklahoma Department of Career and Technology Education 

existing industry training program: Economic impact, return on investment, and customer 
satisfaction. (Doctoral dissertation, Oklahoma State University). Retrieved from 
ABI/INFORM Global. (Publication No. AAT 3181689). 

http://proqucst.um i.com/pqdlink?did=95399987 1 &Fmt=7&clicntld=47297&RQT=309&VNamc 
=PQD 


Dissertation Findings and Conclusions, Sub-System Level 

This dissertation was primarily an assessment of the Existing Industry Training program by 
CareerTech technology centers within the state of Oklahoma. ROI, economic impact, and 
customer satisfaction pertaining to the program were generated from surveys distributed to 
Oklahoma businesses that took part within the training program. The study found a positive rate 
of return within the survey cohort and found a per-dollar ROI by the Oklahoma Department of 
Career and Technical Education on the Existing Industry Training program of nearly 400% 
within local businesses. 

(CS) Snead, M. C. (2004). Moore-Norman Technology Center full-time programs: Income gains 
and economic impacts. Stillwater, OK: Oklahoma State University, Center for Applied 
Economic Research. 

http://www.crossroads.odl.state.ok.us/cdm4/item viewer.php?CISOROOT=7stgovpub&CISOPT 
R=13957&CISOBOX=l&REC=5 

http://spears.okstate.edu/files/documents/caer/Research/MNTC 04 FT Wage Model.doc 
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Study Highlights, Project Level 


Using fiscal year 2003 data, this study provides an estimate of the economic impacts and income 
gains associated with completing a full-time program at the Moore-Norman Technology Center 
affiliated with Oklahoma’s CareerTech system. From the 2,200 students completing full-time 
programs at MNTC, the average individual will add approximately $155,000 in current dollars to 
their lifetime earnings stream. Across the entire cohort, increased income amounts to a $95 
million addition to future income streams with ancillary financial support of $46.2 million to 
other workers statewide. Over the work life of the average program completer, $4.3 million in 
sales and income taxes will be paid ($7,100 per completer), which supports an additional $3.5 
million in indirect sales and income taxes from other Oklahoma workers. 

(CS) Snead, M. C. (2006). Completers of technology center full-time programs: Lifetime income 
gains and the impact on the Oklahoma economy. Stillwater, OK: Oklahoma State 
University, Center for Applied Economic Research. 

http://www.elevatingoklahoma.com/flash/snead full.pdf 
http://www.elevatingoklahoma.com/html/snead summary.pdf 

Study Highlights, Sub-System Level 

Using fiscal year 2002 data, this study provides estimates of the economic impacts of completers 
of full-time CareerTech training programs on the Oklahoma economy. Based on Oklahoma 
Employment Security Commission employer-reported wage data and Census Bureau survey 
data, the economic impact of the state’s 1 1,680 technology center program completers was 
assessed. The average program completer added approximately $371,000 ($152,500 in current 
dollars) to their lifetime earnings stream; across the entire 2002 cohort, this amounts to an 
estimated $4.3 billion ($1.8 billion in current dollars) of which $1.1 billion is expected to be 
earned within the Oklahoma. An additional $990 million is expected in future earnings through 
other workers via multiplier effects and lifetime direct sales and income taxes of $82.6 million 
over the cohort work life (approximately $7,000 per completer) will be generated along with 
multiplier effects of $74 million in current dollar tax revenue by other workers statewide. 

South Carolina 

(CS) University of South Carolina, The Darla Moore School of Business Division of Research. 
(2009). The economic return on investment in South Carolina ’s higher education. 
Columbia, SC: University of South Carolina. 

http://www.che.sc.gov/InfoCntr/HESC Files/EconReturnHigherEd August09.pdf 

Executive Summary, State Level 

The South Carolina Higher Education Study Committee (HESC) sponsored this research in order 
to more fully assess the benefits of higher education within the state and to determine the 
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associated costs of such. This study found that a bachelor’s degree holder in South Carolina will 
earn an average of $2.5 million over his or her lifetime, not accounting for the cost of earning 
such a degree; this amounts to a $ 1 .2 million increase over the average earnings of a high school 
degree recipient and a personal return on college investment of 820%, measured in additional 
lifetime income. The HESC Action Plan sets a target of 29% of the South Carolina workforce 
holding at least a bachelor’s degree by 2030; by achieving this and associated goals, it is 
estimated that this higher education investment will yield a per-dollar annual return of $25.20 in 
economic activity, as measured by gross state product. Attaining HESC Action Plan objectives 
would yield $6.9 billion in new personal income, $7.8 billion in gross state product, and an 
estimated 44,5 14 additional permanent jobs per year with benefits across every region of South 
Carolina. Related study findings specific to South Carolina revealed that 90% of the prison 
population has no college degree; Black students graduate at a much lower rate (1 1.7%) than 
White students (26.8%) at the national level; 8.5% of college graduates versus 37.5% of high 
school graduates lack health insurance; and 5.7% of bachelor’s degree and 33.3% of high school 
graduates receive Medicaid benefits. 

Tennessee 

(CS) Harrison, H. D., Earnest, D., Grehan, L., & Wallace, J. (2006). The economic impact of 
secondary and post-secondary career and technical education in Tennessee. Memphis, 
TN: The University of Memphis, Sparks Bureau of Business and Economic Research. 

http://www.tn.gov/education/cte council/doc/execsummary.pdf 

Project Summary, Sub-System Level 

Assessing the impact of secondary and postsecondary CTE on participants and the economy of 
Tennessee were the primary objectives of this project. The 2001-2002 CTE cohort consisted of 
36,000 graduates and completers, total earnings were for the cohort were broken down by 
program type with 7,900 WIA Career Center graduates earning $1 17.5 million, 13,000 secondary 
CTE graduates earning $99.8 million, 1 1,000 technology center graduates earning $263 million, 
and 4,000 community college CTE graduates earning $98.4 million across several industries. 
Output, labor income, and tax impact figures were also estimated, with a total output impact of 
$851,668,009, a labor income impact of $230,290,266, and a total tax impact of $76,740,782. 
Together these yield an economic impact in excess of $1.1 billion. A specific return on 
investment ratio was also reported with secondary and postsecondary CTE yielding a 1:1.01 ROI 
for Tennessee. CTE expenditures were found to have a cost-benefit ratio of 1 : 1 .99, and by 
combining CTE expenditures and earnings, a cost-benefit ratio of 1:5.37 was determined. 

(CS) Martin, F. (2007). Pellissippi State Technical Community College - A major partner in the 
economic vitality of the Knoxville metropolitan area: An analysis of the college ’s 
economic impact, 2002-2007. Knoxville, TN: Pellissippi State Technical Community 
College. 

http://www.pstcc.edu/departments/institutional research/economic/eis 2002-2007.pdf 
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Executive Summary, Project Level 

This report focused exclusively on the direct economic impact of Pellissippi State Technical 
Community College on Knox and Blount counties in Tennessee. Revenues over the five year 
2002-2007 period totaled in excess of $275 million with external revenues comprising $182 
million and local revenues comprising $93 million. College expenditures of institutional 
revenues within the service area resulted in business volume amounting to $314 million, and in 
excess of 22,000 jobs were either created or sustained by the college. The college’s total 
economic impact within the Knoxville metropolitan area was calculated to be $636 million for a 
total ROI per local dollar of between $6.83 and $7.15. 

Texas 

(CS) Keaver, J. (2008, October 21). HCC highlights its economic impact to area: Leaders say the 
school contributes $4.1 billion yearly. The Houston Chronicle. 

http://www.chron.com/disp/storv.mpl/nb/alief/news/6071396.html 

Article, Project Level 

This article summarizes the findings of an economic impact study of the six-college Houston 
Community College (HCC) district. The timing of the study corresponded with an election where 
community members could vote whether to join the taxation district of the HCC or not. Annual 
economic impact of the HCC upon the Houston metropolitan area was calculated to be $4.1 
billion, with degree completing students earning an additional $577,200 in lifetime earnings or 
$56,500 annually at their career midpoint. 

(CS) King, C. T., & O’Shea, D. (2003). Estimating return-on-investment (ROI) for Texas 
workforce development boards: Lessons learned and next steps. Austin, TX: The 
University of Texas at Austin, Ray Marshall Center for the Study of Human Resources. 

http://www.utexas.edu/research/cshr/pubs/pdf/roi lessons.pdf 

Report, Project Level 

This 2003 report discusses the challenges associated with performing an ROI analysis for 
workforce services within Texas. Data-related problems were discussed as well as the inadequate 
resources budgeted for conducting the analysis, an economic impact estimation study is proposed 
for additional phases of the workforce service evaluation process. 
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(CS) King, C. T., O’Shea, D., Looney, S. E., Redman, C. A., & Holcombe, W. L. (2003). 

Return-on investment ( ROI) estimates for workforce services in Texas, state fiscal year 
2000-2001: Composite workforce development board. Austin, TX: The University of 
Texas at Austin, Ray Marshall Center for the Study of Human Resources. 

http://www.utexas.edu/research/cshr/rmcl/index.php/publications/all-publications/50- 

about.html?catid=6%3Aabout 


Report Executive Summary, State Level 

ROI estimates were generated for workforce services delivered in 18 regions of Texas within this 
study, a preliminary attempt at estimating ROI across multiple workforce funding streams within 
the state. Five and 10-year reasonable first-approximations of net returns to taxpayers for major 
workforce investments in the Composite Workforce Development Board were estimated with a 
five-year return of 600% and a 10-year return of 800%. The per-dollar ROI figures for workforce 
services in 2000-2001 in Texas were calculated to be 6.0 after five years and 8.0 after 10 years. 
Several benefits such as returns for additional years of schooling and savings from reduced 
criminal activity as well as costs association with childcare and program transition were not 
included in the analysis, lending a conservative bias to the findings. 

(CS) King, C. T., Yang, Y., Smith, T. C., & Schroeder, D. G. (2010). Texas workforce 

investments: returns for participants, taxpayers, and society. The Free Library. Austin, 
TX: University of Texas at Austin Bureau of Business Research. 

http://www.thefreehbrary.com/Texas+workforce+investments%3A+returns+for+participants,+ta 
xpayers,+and...-a02304 16376 

Article, State Level 

This article summarizes ROI estimates for comprehensive workforce services delivered through 
local workforce boards in Texas under the Texas Workforce Commission. The state of Texas 
invested approximately $1.1 billion on workforce services in fiscal year 2005, with estimated 
five-year ROI figures of 12%, 29%, and 25% for taxpayers, participants, and society, 
respectively. Ten-year ROI estimates ranged from 25% for taxpayers, 38% for participants, and 
35% for society. Per-dollar ROI in work services figures were detailed, with participant returns 
at $1.63 over five years and $2.74 over 10 with costs incurred of $5,007 per participant. 

Taxpayer returns were found to be $ 1 . 17 over five years and $2.08 over 10 with costs of $ 1 .00 
per participant. Societal returns were reported at $1.52 over five years and $2.58 over 10 with 
costs of $6,527. Methodological constraints and controls are discussed within the article as well 
as recommendations for improving ROI estimates and workforce services. 
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(CS) Norris, D. N., & King, C. T. (1997). Return on investment: A cost-effectiveness measure for 
the Texas workforce system. Austin, TX: The University of Texas at Austin, Center for the 
Study of Human Resources. 

htt p ://www. eric . ed.gov/ERICW ebPortal/ detail?accno=ED4 15384 
Report Abstract, State Level 

This report focuses on the development of an ROI measure to assess the cost effectiveness of 
workforce programs within Texas. The report focuses on methodological challenges and 
measurement constraints specific to performing return on investment analysis at the level of the 
Local Workforce Development Board system in place in Texas. Best practices for performing 
ROI analysis at this level are identified particularly relative to the Job Training Partnership Act 
along with limitations and key issues. 

(CS) Texas Comptroller of Public Accounts. (2008). The economic impact of Texas community 
colleges. In Texas Works: Training and Education for all Texans (pp. 53-59). Austin, 

TX: Texas Comptroller of Public Accounts Data Services Division. 

http://www.window.state.tx.us/specialrpt/workforce/colleges.php 

Report Chapter, Sub-System Level 

This chapter of the Texas Comptroller of Accounts publication, Texas Works: Training and 
Education for All Texans, estimates measures of the economic impact of community and 
technical colleges on the total state economy. The two analyses measure different aspects of this 
economic impact by detaining the money brought into Texas from out of state and the resulting 
impact of earnings from all Texans with associate degrees as well as the specific returns for 
individual students. The 2008 economic output multiplier revealed a 95% return per external 
dollar of investment within Texas community colleges, with a total economic impact of $2. 1 
billion. Components of this economic impact associated with Texas community colleges 
included $836.5 million from federal grants and contracts, $223.6 million from tuition and 
books, and $1.06 billion total out of state money. The second analysis that focused on returns on 
investment for individuals who earn an associate’s degree found that, over 2005-2007, there 
were an average of 57,596 two-year graduates, with 94.1% being in state. The average Texan 
with an associate’s degree was calculated to earn $125,546 more over a career, as a cohort, 
associate’s degree recipients were found to have an average employment rate of 75.7%, total 
lifetime earnings of approximately $5.2 billion, and a total impact on the Texas economy in 
excess of $10 billion. Career-specific ROI calculations were also provided in addition to a 
discussion regarding the associated social benefits of investing in two-year postsecondary 
institutions. 
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Utah 


(CS) Brown, A., & Hoyt, J. E. (2000). The economic impact of Utah Valley State College 1999- 
2000. Orem, UT: Utah Valley State College, Department of Institutional Research and 
Management Studies. 

http://www.cric.cd.gov/ER 1C WcbPortal/detail?accno=ED468 1 39 
Report Abstract, Project Level 

Utah Valley State College (UVSC), located in Utah County, Utah, is a state college composed of 
both a two-year division and an upper baccalaureate degree-granting division. Using the Ryan- 
New Jersey model and a U.S. Department of Commerce multiplier for estimating economic 
impact, this report performs an estimate of the impact of UVSC on the surrounding district over 
1999-2000. UVSC was associated with approximately $153 million of economic activity in Utah 
County, generating 884 full-time and 1,485 part-time jobs along with employment for 796 
students. The calculated per-dollar ROI figure for state investment in UVSC over this period was 
$4.04 within the surrounding community. The report also estimates the economic impacts of 
UVSC-affiliated programs and centers individually. 

(CS) EMSI. (2010). The economic contribution of Salt Lake City College. Moscow, ID: 
Economic Modeling Specialists, Inc. 

http://active.slcc.edu/ir/docs/ec exec sum 0809.pdf 
http://www.slcc.edu/ir/docs/ec main rpt 0809.pdf 

Executive Summary, Project Level 

Investment and economic growth analyses comprise the majority of this socioeconomic impacts 
study of Salt Lake City College (SLCC). Taxpayer rate of ROI was calculated to be 8.0% related 
to SLCC, with the state of Utah saving $3.5 million annually as a result of social benefits derived 
from students attending the two-year institution. Student investments have a $5.50 per-dollar 
ROI in higher future incomes over their working careers. The total economic impact of SLCC on 
Salt Lake County amounted to $1 13 million annually, with $90.4 million consisting of net added 
income from institutional operations. 

Virginia 

(CS) Bolipata, K. (2008, November 19). Gennanna’s impact is examined: Study finds that 
Germanna contributes $241 million to the Lredericksburg region each year. The Free 
Lance-Star. 

http://fredericksburg.eom/News/LLS/2008/l 12008/1 1 192008/426221 
Article, Project Level 
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This article summarizes the findings of an economic impact study of Gennanna Community 
College upon its eight county service area including ROI figures specific to both students and 
taxpayers. The total economic impact of GCC was calculated to be $241.2 million annually, 
amounting to 2.1% of the service area’s annual income. With 95% of GCC students remaining in 
Virginia, contributions to economic growth are further substantiated through decreased state 
costs of approximately $235,900 annually. Institutional per-dollar investment was calculated to 
generate a $3.70 return within the local economy, whereas student investments yielded annual 
income increases of $153 per credit hour completed and $385,200 in additional lifetime earnings 
per degree completed. 

Washington 

(CF) Hollenbeck, K. (2005). On the use of administrative data for workforce development 

program evaluation. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. 

http://research.upiohn.org/cgi/viewcontent.cgi7articleM 027&context=externalpapers&sei- 

redir=l#search=%22Use%20Administrative%20Data%20Workforce%20Development% 
2 OPro gram%20Evaluation%22 

Report Introduction, Methodology, State Level 

This report attempts to detennine whether performance monitoring data such as administrative 
information from the State of Washington can be effectively used within program evaluation, in 
this case adult services provided under the Workforce Investment Act (WIA) through a quasi- 
experimental method. A key objective of the report was to analyze econometric estimation 
techniques relative to program evaluation. Specific results presented are relative to WIA within 
Washington State over the program year July 2000 to June 2001; economic estimator 
methodology is detailed. 

(CS) Career and Technical Education, Washington Workforce Training and Education 
Coordinating Board. CTE: An investment in success. Olympia, WA: Author. 

http://www.wtb.wa.gov/Documents/CTESuccess.pdf 

Report, Sub-System Level 

This report by the Washington Workforce Training and Coordination board details the individual 
and public benefits associated with CTE programs. General return on investment figures are 
provided within the report, comparing the public costs associated with a student completing high 
school CTE or a Skills Center program ($920) with the additional taxes generated by CTE 
completers ($6,600) for an approximate 700% ROI figure. The benefits associated with CTE 
were also calculated at multiple levels from workforce training program data. High school CTE 
graduates were found to earn an average of $840 more annually compared to standard graduates, 
and were also employed at a 6.7% higher rate than standard high school graduates. 

Postsecondary CTE graduates were found to earn $7,700 more annually while being employed at 
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a 13% higher rate than standard high school graduates. Private CTE graduates were found to earn 
an average of $3,500 more annually at a 7.3% greater employment rate than standard high school 
graduates. 

(CF) Hollenbeck, K. M., & Huang, W. J. (2006). Net impact and benefit-cost estimates of the 
workforce development system in Washington state. Kalamazoo, MI: W. E. Upjohn 
Institute for Employment Research. 

http://www.upiohninst.org/publications/tr/tr06-020.pdf 

Report Abstract, State Level 

This report details study estimates of economic impact for 1 1 workforce development programs 
within Washington State. These consisted of WIA Title I-B Adult programs, WIA Title I-B 
Dislocated Worker programs, Community and Technical College (CTC) Job Preparatory 
Training, CTC Worker Retraining, Private Career Schools, Apprenticeships, CTC Adult Basic 
Skills Education, Division of Vocational Rehabilitation programs, Department of Services for 
the Blind programs, WIA Title I-B Youth programs, and secondary career and technical 
education. Estimation techniques and analyses were based on administrative data from the 
various programs as well as the Unemployment Insurance wage record system and TANF. Short- 
term net impacts and earnings impacts for individuals are positive for 9 of the 1 1 programs, 
longer-term employment impacts are positive for all programs, and earning impacts are positive 
for 10. Benefit cost-analyses indicate that future benefits have a positive return on costs for both 
participants and society. ROI figures are program-specific and are provided in table fonnat. 

Wisconsin 

(CS) Christopherson, K. A., & Robinson, M. H. (2001). The socioeconomic benefits generated 
by the Wisconsin Technical College System. Madison, WI: Wisconsin Technical College 
System Board. 

http://www.eric.ed.gov/ERICWebPortal/detail7accnoMiD462091 
Report Abstract, Sub-System Level 

Using the Association for Community College Trustees’ (ACCT) model for measuring the 
economic and social impacts of technical colleges, this report details estimates on the related 
effects of the Wisconsin Technical College System (WTCS). The ACCT model measures four 
types of technical college benefits, including income and job creation, improved graduate 
earnings, social benefits, and returns on investment for taxpayers. Results of the analysis 
revealed that WTCS is associated with $3,852 billion in annual earning for Wisconsin; students 
experience a 20% return on time and money investments. Associated state statistics indicate 
numerous social benefits and savings of $45.5 million, and taxpayers receive a 100% return on 
investments within 2.2 years. 
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National Studies 


As was done with the state-level studies summarized above, national studies were also obtained 
through standard literature review practices, creating individual summaries of major findings and 
key information relevant to ROI analysis for CTE. As with the state studies, each summary was 
then categorized as to type of study and type of ROI analysis, using definitions and parameters 
for each as outlined in this report. Studies in which the type of analysis delineations was 
inconclusive or subjective were initially left undefined and then re-categorized so as to best 
represent the majority of the more objectively defined studies previously identified. It should be 
pointed out that the studies summarized below are listed as national because their focus is such. 
Nevertheless, many of the studies include state, system, and program information, and some use 
such infonnation to draw conclusions on ROI from a national perspective. 

In the summaries that follow, these abbreviations are used: CF = Common Framework, SB = 
Social Benefit, and CS = Case Study. 


(SB) Kelly, P. J., & Jones, D. P. (2005). A New Look at the Institutional Component of Higher 
Education Finance: A Guide for Evaluation Performance Relative to Financial Resources. 
Boulder, CO: National Center for Higher Education Management Systems. 

http://www.higheredinfo.org/analvses/Policy%20Guide%20Dec2005.pdf 

This evaluation focused upon analyzing the ability of higher education institutions to improve 
levels of perfonnance with existing or diminishing financial resources. Maintaining and 
improving access to postsecondary education was assessed using three principle components: 
affordability, a state’s ability to pay, and detennined adequacy of institutional finance. 
Developing methods for assessing institutional finance adequacy relative to institutional 
performance and mission attainment within various types of institutions was a primary objective 
of this report. State-specific institutional ROI information is provided in table format. 

(SB) Karpowitz, D., & Kenner, M. (1995). Education as crime prevention: The case for 

reinstating Pell grant eligibility for the incarcerated. New York, NY: Bard College. 

http://www.inpathways.net/ipcnlibrary/ViewBiblio.aspx7aicU276 

This report cites several official U.S. government publications that support reinstating Pell grant 
eligibility to qualified incarcerated individuals as a means of both reducing recidivism and 
preventing crimes. Extrapolating findings from cost-benefit and ROI analyses yielded projected 
state savings in the millions of dollars as a result of reinstituting inmate Pell grant eligibility. 

(SB) Cresswell, A. M., Burke, G. B., & Pardo, T. A. (2006). Advancing return on investment 
analysis for government IT: A public value framework. Albany, NY : University at 
Albany, SUNY, Center for Technology in Government. 

http://www.ctg.albanv.edu/publications/reports/advancing roi 
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This paper proposes a public ROI assessment framework based upon public value generated 
from investments made in government IT. The public value framework attempts to connect 
actions within government to impacts on public domain stakeholders with subsequent reporting 
of findings and application of results. 

(CS) Kemple, J. J., & Willner, C. J. (2008). Career academies: Long-term impacts on labor 

Market outcomes, educational attainment, and transitions to adulthood. New York, NY: 
MDRC. 

http://www.mdrc.org/publications/482/overview.html 

This report by MDRC assesses how career academies, as an educational reform concept, have 
the ability to influence student labor market returns and postsecondary educational attainment. 
The evaluation surveyed a diverse student population from nine high schools located throughout 
the United States through a randomized research design. Specific ROI findings include an 1 1% 
average annual earnings gain ($2,088) among students participating in career academies 
compared to a control group; this represents a $16,704 increase in total earnings over eight years. 
Labor market impacts were primarily among young men with real earnings increasing by 17% 
($3,731) among this demographic; approximately $30,000 over eight years. Career academies 
did not appear to be more effective in encouraging participation in postsecondary education than 
traditional high schools; however, rates of independent living and marriage were higher among 
the career academy participants. 

(CS) Anderson, M. C. (2004). Case study on the return on investment of executive coaching. 
Johnston, IA: Metrix Global LLC. 

http://www.workplacecoaching.com/pdf/CasestudvonROLpdf 

This executive coaching program evaluation details the business benefits and overall ROI figures 
for financing executive coaching initiatives. A 529% ROI calculation was assessed along with 
significant ancillary benefits associated with the executive coaching program. Qualitative and 
quantitative evaluation data was obtained through the administration of a survey instrument to 
program participants. ROI calculations were based upon program costs and derived financial 
business benefits with the most significant returns being found among executives with client or 
human resource responsibilities. 

(CS) Haley, D. (2006). Investing in people: Measuring the economic impact of a job training 
program in an area with high structural unemployment through cost-benefit 
analysis. (Doctoral dissertation, Nova Southeastern University). Retrieved from 
ABI/INFORM Global. (Publication No. AAT 3228634). 

http://proquest.umi.com/pqdweb7didM23242446 l&sid=2&Fmt=2&clientId=47297&RQT=309 
&VName=PQD 
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The McAllen Self-Sufficiency Project (MSSP) is a job training program within the Rio Grande 
Valley that is designed to assist residents within the City of McAllen, TX, in attaining the 
education and training necessary for employment and financial independence. A community 
partnership between a two-year college, education center, and workforce development team, 
MSSP has attained an 85.4% retention/graduation rate along with 88.9% employment in the 
greater McAllen area. Operating on a city investment of $2,451 million derived from sales tax, 
MSSP completers have shown average annualized wage increases from $7,296.24 to $29,503.36. 
The calculated program ROI, adjusted for earnings and emigration, was 1 1 1%, a $2. 1 1 return per 
dollar invested in MSSP, for a total net benefit of $5. 1 12 million for the area. 

(CS) Dadayan, L. (2006). Measuring return on government IT investments. Albany, NY : 
University at Albany, SUNY, Center for Technology in Government. 

http://www.ctg.albanv.edu/publications/ioumals/ecite 2006 roi 

Dadayan employs a comparative analysis approach to investigating the various models and 
methodologies developed for evaluating returns on investment in the public sector relative to 
information technology investments specifically. This paper focuses exclusively on infonnation 
technology evaluation within the public sector, assesses several trends within the area, and 
qualitatively compares and contrasts seven different public return on investment models (Social 
Return on Investment Model SROI, Balanced E-Govemment Index BEGIX, Value Measuring 
Methodology VMM, Public Sector Value Model PSV, Performance Reference Model PRM, 
Interchange of Data between Administrations Value of Investment IDA VOI, and Demand and 
Value Assessment Methodology DAM/VAM). 

(SB) Jacobson, L., & Mokher, C. (2009). Pathways to boosting the earnings of low-income 

students by increasing their educational attainment. Washington, DC: Hudson Institute 
Center for Employment Policy. 

http://www.hudson.org/files/publications/Pathwavs%20to%20Boosting.pdf 
http ://www. eric . ed.gov/ERICW ebPortal/ detail?accno=ED5 0407 8 

The identification of educational pathways to high-paying careers that could in turn improve 
social mobility was a primary objective behind this study. Associated with the identification of 
pathways was an assessment of how much of an effect students’ educational preparation and 
performance had upon making a successful transition. A cohort of 144,545 ninth-grade students 
from 1996 in Florida was analyzed using comprehensive high school, postsecondary, and 
workforce data. Major findings from this report included higher earnings being correlated with 
postsecondary degrees, particularly within professional and health-related fields. STEM 
academic concentrations were found to be the most lucrative; high school preparation and 
performance were found to be strong predictors of postsecondary persistence; and low-income 
students were determined to be underrepresented in all student populations. As a population, they 
earn approximately 10% less than other student groups, an effect attributed to postsecondary 
persistence and academic concentration selection. 
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(SB) Courtright, S. H., & Fry, C. G. (2007). Public rates of return on higher education 
investments, by state. Journal of Teaching & Learning, 4(f), 13-26. 


http:// j ournals . cluteonline . com/index ■php/TLC/article/ view/ 1553 

Differential tax revenues collected from college-educated citizens versus high school-educated 
citizens were used as the basis for this return on public investments in higher education analysis. 
Courtright and Fry found that the majority of states had an appreciable rate of return on higher 
education investments based solely on differential tax revenue; the authors concluded that 
ancillary social benefits and other factors most likely influence state financing and public support 
for funding higher education systems. State-specific figures and calculations for differential tax 
revenues are provided in table fonnat. 

(CS) Erdogmus, H., Favaro, J., & Strigel, W. (2004). Guest editors’ introduction: Return on 
investment. IEEE Software, 21(3), 18-22. Retrieved from ABI/INFORM Global. 
(Document ID: 1750758701). 

http://proquest.umi.com/pqdhnk?did=1750758701&Fmt=6&chentId=9580&RQT=309&VName 

=PQD 


This article details ROI analysis as it pertains to the software industry with specific emphasis 
placed on applications within economic and strategic analysis. Erdogmus, Favaro, and Strigel 
divide ROI analysis relative to the software industry into four categories: business strategy, 
valuation, cost-benefit analysis, and metrics. Each category is explained in detail as different 
approaches toward analyzing and creating economic value in the software industry. 

(CS) Glover, R. W., Long, D. W., Haas, C. T., & Alemany, C. (1999). Return-on-investment 
(ROI) analysis of education and training in the construction industry. Austin, TX: The 
University of Texas at Austin, Center for Construction Industry Studies. 

http://www.utexas.edu/research/cshr/rmcl/index.php/publications/all-publications/237-retum- 
on- investment. html?catid= 1 0%3 Aabout 


This report conducts a comprehensive review of ROI analyses as they pertain to education and 
training; the authors make recommendations relative to applications within the construction 
industry and pertinent workforce development initiatives. Glover, Long, Haas, and Alemany 
identified two main approaches to ROI analysis: the business practitioner approach, rooted in 
logic, simplicity, transparency, and practicality; and an academic approach, emphasizing 
scientific rigor and replicability. Employer, trainee, and government or public/taxpayer 
perspectives for return in investment analysis are detailed as well as efficiency/effectiveness 
applications within construction industry workforce development. 

(SB) National Association of State Directors of Career Technical Education Consortium. (2010). 

Return on investment in CTE. Silver Spring, MD: Author. 
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http://www.careertech.org/legislation/briefs-papers.html 

The individual and societal benefits associated with CTE within three states are summarized 
within this document, highlighting positive returns on investment associated with funding and 
support for CTE programs. A 2006 study of Oklahoma CareerTech program graduates found 
individual benefits relative to high school graduates, including 20% increased wages and $4,100 
annual income increases; state benefits totaled $2.4 billion both directly and indirectly with the 
majority attributed to increased tax revenues from wages. A 2006 report on Tennessee CTE 
initiatives found state benefits consisting of a cost/benefit ratio of 1 : 1 .99, a fiscal turnover ratio 
of 1:1.01, and a total ROI ratio of 1:5.37 for state CTE investments. A 2006 report on 
Washington State CTE programs found state benefits in decreased TANF, food stamp, and 
Medicaid eligibility with individual benefits for CTE program completers including hourly wage 
increases with cumulative annual earnings impacts. 

(SB) Alliance for Excellent Education. (2006). Saving futures, saving dollars: The impact of 
education on crime reduction and earnings. Washington, DC: Author. 

http://www.all4ed.org/files/archive/publications/SavingFutures.pdf 

This article focuses on the social benefits derived from improving high school completion rates, 
primarily reducing crime while increasing earnings. The Alliance for Excellent Education 
analyzed the impact of increasing educational attainment on crime reduction by calculating the 
annual crime -related savings attributed to decreased incarceration costs associated with a 5% 
increase in the male graduation rate. State specific savings would vary significantly but an 
estimated $2.8 billion in additional annual earnings would be generated by a 5% graduation rate 
increase. State-specific calculated annual crime-related savings, additional annual earnings, and 
total state economy benefit are provided in table format. 

(SB) Institute for Higher Education Policy. (2005). The investment payoff: A 50-state analysis of 
the public and private benefits of higher education. Washington, DC: Author. 

http://www.ihep.org/Publications/publications-detail.cfm?id=43 

http://www.ihep.org/assets/Files/publications/g-l/InvestmentPavoff.pdf 

The Institute for Higher Education Policy established the articulation of higher education- 
associated benefits on a 50-state basis as the principle objective behind this report; benefits were 
placed into one of four categories: public economic, private economic, public social, or private 
social. Replicating previously researched national benefits of higher education on an individual 
state basis was deemed necessary in order to serve as a rationale for maintaining state funding for 
postsecondary education in the current difficult economic climate. Six measurable indicators 
were quantified within all states: private economic by higher personal income and lower 
unemployment, public economic by decreased reliance on public assistance, private social 
through better health, and public social through increased volunteerism and increased voting 
participation. The additional earning value associated with earning a college degree served as the 


56 



basis for many of the calculations and statistics with further derivations for partial completion. 
State-specific ROI statistics are provided in table format. 

(SB) Wolfle, D., & Smith, J. G. (1956). The occupational value of education for superior high- 
school graduates. The Journal of Higher Education, 27(4), 201-212, 232. 

http://www.istor.org/stable/1977699 

This study assessed the longitudinal difference in occupational distribution of similarly qualified 
high school graduates who chose to pursue postsecondary education versus those who did not. 
Wolfe and Smith, in association with the Commission on Human Resources and Advanced 
Training, administered surveys to a three-state cohort of high school graduates deemed prepared 
and qualified for college work from approximately 20 years prior. College graduates were found 
to earn an average of $1,100 to $2,500 more than those high school completers who were 
prepared for college but did not attend, controlling for both assessed intelligence and class ra nk . 
The authors identified the completion of postsecondary education as the primary factor 
influencing annual earnings over both class rank and intelligence scores in this cohort. 

(SB) Harmon, C., Oosterbeek, H., & Walker, I. (2003). The returns to education: 
Microeconomics. Journal of Economic Surveys, 77(2), 115-155. 

http://facultv.smu.edu/millimet/classes/eco7321/papers/hannon%20et%20al%202003.pdf 
http://ideas.repec.Org/a/bla/iecsur/v 1 7y2003i2p 1 1 5- 1 56.html 

Approaching education as an individual decision to invest in human capital, this article details 
the methodologies and processes associated with accurately estimating the rate of return on such 
investments. Harmon, Oosterbeek, and Walker conclude from multiple regression analyses 
across several datasets that personal education investments are almost unilaterally positive and 
outpace comparative investments as well. United Kingdom microdata provided rates of return of 
between 7% and 9% for men and 9% and 1 1% for women. 

(SB) Sianesi, B., & Reenen, J. V. (2003). The returns to education: Macroeconomics. Journal of 
Economic Surveys, 77(2), 157-200. 

http://onlinelibrary.wilev.eom/doi/10.l 1 1 1/1467-6419.00192/pdf 
http://ideas.repec.Org/a/bla/iecsur/vl7v2003i2pl57-200.html 

This article is primarily a critical literature review focusing on human capital’s impact on 
macroeconomic performance. Sianesi and Van Reenen found that existing literature supports the 
assertion that human capital increases productivity whereas applications and extensions to gross 
domestic product (GDP) are less supportive. Major literature findings include a one-year 
increase in average population education corresponding with an increase in output per capita of 
3% to 6% amid 1% faster growth. Educational economic impact was found to be dependent upon 
a country’s level of development; emphasis was placed on the role of tertiary education within 
Organisation for Economic Cooperation and Development (OECD) countries. Education was 
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found to make additional indirect contributions to both economic growth and development. 
Article-specific literature review ROI analysis figures are provided in addition to application 
analysis statistics. 

(SB) Vawda, A. Y. (2003). Who benefits from public education expenditures? Economic Affairs, 
23, 40-43. doi: 10.1 1 1 1/1468-0270.00399 

http://onlinelibrarv.wilev.com/doi/10. Ill 1/1468-0270.00399/abstract 

Vawda assesses whether public expenditures on education have been both effective and equitable 
related to the socioeconomic statuses of populations described within this article. Analysis 
revealed systemic inequities regarding educational funding, with poor income groups being 
underrepresented in post-primary education. Across selected countries and drawing upon 
existing research, private rates of return to schooling by family background, public expenditure 
distribution by income, and enrollment in primary and tertiary public education systems were 
considered in assessing relative educational equity. Private rates of return to schooling by family 
background are provided in table format. 

(CS) Holzer, H. J. (2009). Workforce development as an antipoverty strategy: What do we 
know? What should we do? Focus, 26(2), 62-68. 

http://www.irp.wisc.edu/publications/focus/pdfs/foc262k.pdf 

http://mitsloan.mit.edu/iwer/pdf/0809-holzer.pdf 

Holzer approaches systemic labor market problems for disadvantaged populations by 
highlighting the disconnect between the importance that is almost universally placed on 
developing workforce skills by economists and policy analysts and the lack of support associated 
with workforce development initiatives, particularly with regard to population economic 
outcomes. Trends in federal funding, the rapidly changing labor market, and political 
divisiveness related to workforce development were analyzed in developing the conclusion that 
substantive evidence for decreasing support of such initiatives simply does not exist. Cost- 
effectiveness rates of return specify the Earned Income Tax Credit (EITC) at $ 1 :$ 1 .25 while the 
Job Training Partnership Act (JTPA) exhibited a 200% return over a five year period. New 
training approaches are also highlighted, including regional sectoral training programs, career 
ladder programs, and incumbent worker programs. Participation rates are detailed; specific return 
on investment calculations are limited to EITC and JTPA. 
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(SB) Alliance for Excellent Education. (2010). The economic benefits from halving the dropout 
rate. Washington, DC: Author. 

http://www.all4ed.org/liles/NationalMSA leb.pdf 

This publication highlights the predicted economic and social benefits associated with halving 
the national high school dropout rate. Using a 2008 cohort of dropout students derived from 45 
metropolitan areas, the economic benefits of 300,000 additional students graduating were 
calculated and quantified. Key projections included $4.1 billion in increased earnings; $2.8 
billion in additional spending; increased home sales of $10.5 billion; 30,000 new jobs created; 
$536 million in additional tax revenue; increased human capital; and total economic growth of 
$5.3 billion. 
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Appendix B 

Program of Study Design Framework and Components 


PROGRAM OF STUDY DESIGN FRAMEWORK 

1 ircfrit el snip n t ciapreteitm, structure! iiproiet hr Mrentp iciiemc n4 ttrttt in uhn/ i/kidii 
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THE 10 PROGRAM OF STUDY (POS) COMPONENTS 
1. LEGISLATION AND POLICIES 

Federal, state, and local legislation or administrative policies promote POS development and 
implementation. 

Effective legislation and policies should: 

• Provide for state and/or local funding and other resources, such as professional 
development and dedicated staff time, for POS development. 

• Establish formal procedures for the design, implementation, and continuous improvement 
of POS. 

• Ensure opportunities for any secondary student to participate in a POS. 

• Require secondary students to develop an individual graduation or career plan. 

• Provide resources for long-term sustainability of POS. 
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2. PARTNERSHIPS 


Ongoing relationships among education, business, and other community stakeholders are central 
to POS design, implementation, and maintenance. 

Collaborative partnerships should: 

• Create written memoranda of understanding that elaborate the roles and responsibilities 
of partnership members. 

• Conduct ongoing analyses of economic and workforce trends to identify statewide (or 
regional) POS to be created, expanded, or discontinued. 

• Link into existing initiatives that promote workforce and economic development, such as 
sector strategies and other activities supported by the Workforce Investment Act. 

• Identify, validate, and keep current the technical and workforce readiness skills that 
should be taught within a POS. 

3. PROFESSIONAL DEVELOPMENT 

Sustained, intensive, and focused opportunities for administrators, teachers, and faculty foster 
POS design, implementation, and maintenance. 

Effective professional development should: 

• Support the alignment of curriculum from grade to grade (9-12) and from secondary 
to postsecondary education (vertical curriculum alignment). 

• Support the development of integrated academic and career and technical curriculum 
and instruction (horizontal curriculum alignment). 

• Ensure that teachers and faculty have the content knowledge to align and integrate 
curriculum and instruction. 

• Foster innovative teaching and learning strategies (see #9 below). 

4, ACCOUNTABILITY AND EVALUATION SYSTEMS 

Systems and strategies to gather quantitative and qualitative data on both POS components and 
student outcomes are crucial for ongoing efforts to development and implement POS. 
Well-designed accountability and evaluation systems should: 

• Include the “10 Essential Elements of A State Longitudinal Data System” identified by 
the Data Quality Campaign. 17 

• Provide for administrative record matching of student education and employment data 
(i.e., Unemployment Insurance (UI) wage records). 

• Yield valid and reliable data on key student outcomes (indicators) referenced in Perkins 
and other relevant federal and state legislation. 

• Provide timely data to evaluate and improve the effectiveness of POS. 

17 The 10 elements are: (1) statewide student identifier; (2) student-level enrollment data; (3) student-level test data; 
(4) information on untested students; (5) statewide teacher identifier with a teacher-student match; (6) student-level 
course completion (transcript) data; (7) student-level SAT, ACT, and Advanced Placement exam data; (8) student- 
level graduation and dropout data; (9) ability to match student-level P-12 and higher education data; and (10) a state 
data audit system. 
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5. COLLEGE AND CAREER READINESS STANDARDS 


Content standards that define what students are expected to know and be able to do to enter and 
advance in college and/or their careers comprise the foundation of a POS. 

Rigorous college and career readiness standards should: 

• Be developed and continually validated in collaboration with secondary, postsecondary, 
and industry partners. 

• Incorporate essential knowledge and skills (i.e., academic skills, communication, and 
problem-solving), which students must master regardless of their chosen career area or 
POS. 

• Provide the same rigorous knowledge and skills in English and mathematics that 
employers and colleges expect of high school graduates. 

• Incorporate industry-recognized technical standards that are valued in the workplace. 

• To the extent practicable, be internationally benchmarked so that all students are prepared 
to succeed in a global economy. 

6. COURSE SEQUENCES 

Non-duplicative sequences of secondary and postsecondary courses within a POS ensure that 
students transition to postsecondary education without duplicating classes or requiring remedial 
coursework. 

Well-developed course sequences should: 

• Map out the recommended academic and career and technical courses in each POS. 

• Begin with introductory courses at the secondary level that teach broad foundational 
knowledge and skills that are common across all POS. 

• Progress to more occupationally-specific courses at the postsecondary level that provide 
knowledge and skills required for entry into and advancement in a chosen POS. 

• Offer opportunities for students to earn postsecondary credit for coursework taken during 
high school. 

7. CREDIT TRANSFER AGREEMENTS 

Credit transfer agreements provide opportunities for secondary students to be awarded 
transcripted postsecondary credit, supported with fonnal agreements among secondary and 
postsecondary education systems. 

Well-development agreements: 

• Provide a systematic, seamless process for students to earn college credit for 
postsecondary courses taken in high school, transfer high school credit to any two- and 
four-year institution in the state that offers the POS, and transfer credit earned at a two- 
year college to any other two- or four-year institution in the state that offers the POS. 

• College credit should be automatically transcripted at the college for high school students 
so that they can transfer seamlessly into the postsecondary portion of a POS without the 
need for additional paperwork or petitioning for credit. 
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• Describe the expectations and requirements for, at a minimum, teacher and faculty 
qualifications, course prerequisites, postsecondary entry requirements, location of 
courses, tuition reimbursement, and credit transfer process. 

8. GUIDANCE COUNSELING AND ACADEMIC ADVISEMENT 

Guidance counseling and academic advisement help students to make informed decisions about 
which POS to pursue. 

Comprehensive guidance counseling and academic advisement systems: 

• Are based on state and/or local guidance and counseling standards, such as the National 
Career Development Guidelines . 18 

• Ensure that guidance, counseling, and advisement professionals have access to up-to-date 
information about POS offerings to aid students in their decision making. 

• Offer information and tools to help students learn about postsecondary education and career 
options, including prerequisites for particular POS. 

• Offer resources for students to identify their career interests and aptitudes and to select 
appropriate POS. 

• Provide information and resources for parents to help their children prepare for college 
and careers, including workshops on college and financial aid applications. 

• Offer Web-based resources and tools for obtaining student financial assistance. 

9. TEACHING AND LEARNING STRATEGIES 

Innovative and creative instructional approaches enable teachers to integrate academic and 
technical instruction and students to apply academic and technical learning in their POS 
coursework. 

Effective teaching and learning strategies should: 

• Be jointly led by interdisciplinary teaching teams of academic and career and technical 
teachers or faculty. 

• Employ contextualized work-based, project-based, and problem-based learning 
approaches. 

• Incorporate team-building, critical thinking, problem-solving, communication skills, such 
as through the use of career and technical student organization (CTSO) activities. 

10. TECHNICAL SKILLS ASSESSMENTS 

National, state, and/or local assessments provide ongoing information on the extent to which 
students are attaining the necessary knowledge and skills for entry into and advancement in 
postsecondary education and careers in their chosen POS. 

Well-developed technical skills assessments: 

• Measure student attainment of technical skill proficiencies at multiple points during a 
POS. 


18 See http://cte.ed.gov/acm/ncdg/ncdg what.htm . 
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• Employ industry-approved technical skill assessments based on industry standards, where 
available and appropriate. 

• Employ State-developed and/or approved assessments where industry-approved 
assessments do not exist. 

• Incorporate performance-based assessment items, to the greatest extent possible, where 
students must demonstrate the application of their knowledge and skills. 
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Appendix C 

Connecting the 10 POS Components 
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Note. The above graphic was developed by Jill Kroll, Education Research Consultant, Office of 
Career and Technical Education, Michigan Department of Education. Email: 
KrollJl@Michigan.gov . 
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